Today we faced an important issue with MySQL 5.5
Server stalled after aborting a TRUNCATE on a big table. Yes, this is bad to abort a TRUNCATE But this is not the issue I want to speak about. Server crashed, that’s a point. mysqld process restarted automatically, and started loading data into memory (massive use of InnoDB tables). This server has 128GB of memory, so buffer pool was set to 100GB (yes, that’s quite huge). Data set is ~ 100GB. After a few minutes, server crashed again, but this time complaining about memory.
Today we faced an important issue with MySQL 5.5
Il y a quelques jours, j’ai découvert G-WAN, un nouveau serveur web “révolutionnaire” d’après son auteur. Ce dernier annonce des performances assez hallucinantes, même face à Apache, NginX ou Lighttpd qui sont les serveurs web les plus connus.
Ce qui m’a vraiment étonné, c’est la différence de performance annoncée :
J’estime maîtriser assez bien Apache ou NginX, et je m’étonne évidemment de telles différences de performances. J’ai donc décidé d’enquêter ! Continue reading “Tests approfondis de G-WAN” »
A few months ago, I started some tests with RRDTool and a data source in MySQL. At that time, I just saw that performance was awful but did not investigate further. Now is time to check why, and if any improvements could be made. Continue reading “RRDTool with MySQL” »
I had a very old server on a 32 bits platform. A few days ago, I bought a new one with a 64bits platform.
I was using Smokeping on this server, and I wanted to keep my old data. If you copy .rrd files in place, you’ll get an error :
This RRD was created on another architecture
That’s because RRD file structure changed based on the architecture you have. Yes, that’s silly but this is the way it is.
To convert files, you have to dump data to an XML file, sync it on the new server, and reimport it.
This is how to make it with justa few lines of bash :
cd /path/to/rrd/files; for i in $(find . -name '*.rrd'); do rrdtool dump $i > $i.xml ; done rsync -av . my.server.ip:/path/to/new/dir/rrd/
And on the new server :
cd /path/to/new/dir/rrd for i in `find . -name '*.xml'`; do rrdtool restore --force-overwrite $i `echo $i |sed s/.xml//g`; done
A customer asked me to copy a whole database from one mysql server to another.
A few years ago, I would go with the classic mysqldump + import solution, but it is very slow, especially the import part (because MySQL insert buffer is monothread). One can also use mysqlimport (LOAD DATA INFILE), but it is still quite slow… When using a standard SQL dump, I measured a speed of 1MB/sec for reimport … Quite long if you have gigabytes of data !
So I tested xtrabackup for this. This tool is already in heavy tests internally but to my mind, is not quite ready for production. But let’s try this for this specific task of migrating a database.
First, you need to install xtrabackup and all other binaries included (especially xbstream). You’ll also need at least version 5.5.25 on remote host (you’ll see why). And last but not least, InnoDB must run with
In this blog post, I’ll name “server A” the source, and “server B” the destination.
My SQL data is stored in /srv/mysql.
On server B, create a destination folder, for example /tmp/test
This is because you need to import data to a temporary directory, where there is .
On server A, launch the following command :
time innobackupex --export --databases 'mydb' --no-lock \ --stream=xbstream --tmpdir=/tmp --use-memory=128MB \ --user=backup --password=XXXXX --parallel=4 /srv/mysql \ | ssh email@example.com "xbstream --directory=/tmp/test -x"
A few explanations :
- –export : add specific data that would be useful for reimport
- –databases : specify database to copy.
- –no-lock : by default, a FLUSH TABLES WITH READ LOCK is emitted, to ensure the whole backup is consistant. I chose not to use it, as I do not care about binary log position of the backup (used in replication).
- –stream=xbstream : use xbstream as streaming software (more powerful than tar)
- –use-memory : memory that can be used for several tasks
- –parallel=4 : dump 4 tables in parallel
- I pipe stream to ssh and execute xbstream on the remote host.
- I do not use –compress as network is not a limit in my case.
Dump is complete ! I achieved a rate of 11.3 MB/sec with this : far better than mysqldump / mysqlimport ! I’m sure we can be faster by tuning a few things.
Take care of file owner : copy was done with root, but I’m sure your mysql data is owned by someone else (or it should be !).
Then, prepare data for export :
xtrabackup_55 --prepare --export --target-dir=/tmp/test
This will “prepare” data (applying innodb log …), and especially will create .exp files that contains table metadata. This is mandatory for import in the next step !
We can see xtrabackup working :
[...] xtrabackup: export option is specified. xtrabackup: export metadata of table 'mydb/performer' to file `./mydb/performer.exp` (1 indexes) xtrabackup: name=PRIMARY, id.low=443, page=3 [...]
Ok, now we need to create the new database, and import scheme.
On server B, do :
CREATE DATABASE mydb;
We also need a temporary database (you’ll see why later …)
CREATE DATABASE test;
Then, we have two choices :
- create all tables, discard all tablespaces, and import all
- do the same operation, but for each table
I first thought that there were a bug with method 1 (https://bugs.launchpad.net/percona-server/+bug/1052960) but it appears this bug is also hitting if you do method 2. Nevertheless, I’ve created a script that handle method 2 all by itself.
NOTE THAT YOU NEED Percona Server 5.5.25 v27.1 if you don’t want to hit the bug I was talking about. This bug crashes MySQL, and leaves it in a state where it cannot start again … You’ve been warned.
First, let’s dump all tables schemas. On server A, do :
mysqldump --routines --triggers --single-transaction --no-data \ --host=serverA --user=root --password=xxx \ --result-file=schemas.sql mydb;
And import it ON THE TEMPORARY DATABASE :
cat schemas.sql |mysql -uroot -hSERVER_B -p test
Execute following statement in MySQL (server B) to tell MySQL that we will import data :
SET GLOBAL innodb_import_table_from_xtrabackup = 1; /** If you have a version < 5.5.10, despite what I just * said about minimum version required, query is : **/ SET GLOBAL innodb_expand_import = 1;
Then, let’s use following bash script : http://www.olivierdoucet.info/blog/wp-content/uploads/2012/09/expand_import.sh.txt
A few explanations :
For each table, we do the following :
ALTER TABLE xxx DISCARD TABLESPACE; mv xxx.ibd xxx.exp /srv/mysql/mydb; ALTER TABLE xxx IMPORT TABLESPACE;
This bash script is using .my.cnf file in your user dir for credentials (or other default values). Please ensure you have access to the destination database with these credentials.
ALL STEPS ARE REQUIRED. If you miss something (the set global, chown, chmod …) you’ll probably get an error (like ‘got error -1 from storage engine’). At this point, you’d better start the process over (and drop the destination database, which is incomplete).
If import works, you can see lines like this in log :
[...] InnoDB: Import: The extended import of mydb/mytable is being started. InnoDB: Import: 2 indexes have been detected. InnoDB: Progress in %: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 done. [...]
Note that when importing huge InnoDB tables, there is (was) a lock on the dictionary while scanning the ibd file (the data file), so it may lock the whole server … If operation took too long, server crashed. This is bug https://bugs.launchpad.net/percona-server/+bug/684829
This bug has been fixed since, and as you need at least version 5.5.25, you should not hit this problem.
This method is 10 times faster than mysqldump / mysqlimport. But As you can see, there are huge risks and still bugs remaining. The dump part is really safe, so I would recommend you to first test the import on a dev server before doing this in production.
Xtrabackup is really an amazing tool, but it still suffers somme nasty bugs. I’m sure I’ll use it in production in a few months, when it will be perfectly stable for all tasks.
Another major bug in XFS :
After a power failure, server was back but shows some files with size 0 (when I was sure this cannot be possible).
using ‘du -sh’ on this folder shows that total size was > 0 …
The bug was already reported on XFS mailing list here :
If you have many files to recover, procedure is really a pain. That’s why I created a small PHP file that do the trick.
i’m not responsible if you destroy your data But this script worked well for me.
Today I hit a limit on XFS.
A task creating thousands of files was failing with this error :
No space left on device
Even if I still have plenty of disk space :
Filesystem Size Used Avail Use% Mounted on /dev/mapper/backup-logs 2.0T 1.6T 485G 77% /srv/logs
Filesystem used is XFS.
If you also have this bug, fix is very easy : you need to remount the filesystem with “inode64″ option. This is because XFS, by default, only use the first terabyte of data to store inodes. If you create many files, this limit is reached. By adding the mount option ‘inode64′, you give XFS permission to use the whole device to create inodes.
Only drawback is bugs with very old softwares, especially over NFS. This is really a rare situation.
Les développeurs de PHP ont choisis de se focaliser sur les performances plutôt que sur les nouvelles fonctionnalités sur la version 5.4 ; mais cette version est également un grand saut sur beaucoup de fonctions dépréciées, qui sont maintenant totalement supprimées.
La migration de la version 5.3 vers la 5.4 est donc une étape délicate (bien plus que de 5.2 vers 5.3), et de nombreux points sont à vérifier avant de se lancer dans l’opération.
PHP 5.2 n’est plus maintenu depuis janvier 2011, il devient donc tant de migrer sur la version 5.3 (surtout que dorénavant, la 5.4 est sortie en stable …).
Mais une migration doit toujours se préparer, afin qu’elle se passe le mieux possible sans grosse interruption de service. Voici donc une présentation des incompatibilités entre ces deux versions.
J’ai profité de l’été pour refaire la quasi intégralité de mon unité centrale. J’en profite pour vous détailler le montage des pièces.