High Performance MySQL [2 ed.] 0596101716, 9780596101718

Table of contents :
Table of Contents......Page 7
Foreword......Page 11
Preface......Page 13
Building a Solid Foundation......Page 14
Scaling Upward After Making Changes......Page 15
Miscellaneous Useful Topics......Page 16
Software Versions and Availability......Page 17
Using Code Examples......Page 18
How to Contact Us......Page 19
From Peter......Page 20
From Arjen......Page 21
From Jeremy......Page 22
From Derek......Page 23
MySQL’s Logical Architecture......Page 25
Connection Management and Security......Page 26
Concurrency Control......Page 27
Lock Granularity......Page 28
Table locks......Page 29
Transactions......Page 30
Isolation Levels......Page 32
Deadlocks......Page 33
Transactions in MySQL......Page 34
Mixing storage engines in transactions......Page 35
Multiversion Concurrency Control......Page 36
MySQL’s Storage Engines......Page 38
Storage......Page 40
MyISAM features......Page 41
Compressed MyISAM tables......Page 42
The InnoDB Engine......Page 43
The Memory Engine......Page 44
The Federated Engine......Page 45
The NDB Cluster Engine......Page 46
The PBXT (Primebase XT) Engine......Page 47
Selecting the Right Engine......Page 48
Considerations......Page 49
Logging......Page 50
Order processing......Page 51
CD-ROM applications......Page 52
Storage Engine Summary......Page 53
CREATE and SELECT......Page 54
Finding Bottlenecks: Benchmarking and Profiling......Page 56
Benchmarking Strategies......Page 57
What to Measure......Page 58
Benchmarking Tactics......Page 61
Designing and Planning a Benchmark......Page 62
Getting Accurate Results......Page 63
Running the Benchmark and Analyzing Results......Page 65
Full-Stack Tools......Page 66
Single-Component Tools......Page 67
http_load......Page 68
The sysbench CPU benchmark......Page 70
The sysbench file I/O benchmark......Page 71
The sysbench OLTP benchmark......Page 73
Other sysbench features......Page 74
dbt2 TPC-C on the Database Test Suite......Page 75
MySQL Benchmark Suite......Page 77
Profiling......Page 78
How and what to measure......Page 79
A PHP profiling example......Page 80
MySQL Profiling......Page 87
Logging queries......Page 88
Finer control over logging......Page 89
How to read the slow query log......Page 90
Log analysis tools......Page 92
Profiling a MySQL Server......Page 94
Profiling Queries with SHOW STATUS......Page 95
SHOW PROFILE......Page 98
Other Ways to Profile MySQL......Page 99
Troubleshooting MySQL Connections and Processes......Page 100
Advanced Profiling and Troubleshooting......Page 102
Choosing Optimal Data Types......Page 104
Whole Numbers......Page 106
Real Numbers......Page 107
VARCHAR and CHAR types......Page 108
BLOB and TEXT types......Page 110
Using ENUM instead of a string type......Page 111
Date and Time Types......Page 114
Bit-Packed Data Types......Page 115
Choosing Identifiers......Page 117
Indexing Basics......Page 119
B-Tree indexes......Page 121
Hash indexes......Page 125
Isolate the Column......Page 130
Prefix Indexes and Index Selectivity......Page 131
Clustered Indexes......Page 134
Comparison of InnoDB and MyISAM data layout......Page 137
Inserting rows in primary key order with InnoDB......Page 141
Covering Indexes......Page 144
Using Index Scans for Sorts......Page 148
Packed (Prefix-Compressed) Indexes......Page 150
Redundant and Duplicate Indexes......Page 151
Indexes and Locking......Page 153
Supporting Many Kinds of Filtering......Page 155
Avoiding Multiple Range Conditions......Page 157
Optimizing Sorts......Page 159
Updating Index Statistics......Page 160
Reducing Index and Data Fragmentation......Page 162
Normalization and Denormalization......Page 163
Pros and Cons of a Denormalized Schema......Page 164
A Mixture of Normalized and Denormalized......Page 165
Cache and Summary Tables......Page 166
Counter tables......Page 168
Speeding Up ALTER TABLE......Page 169
Modifying Only the .frm File......Page 170
Building MyISAM Indexes Quickly......Page 172
The Memory Storage Engine......Page 173
The InnoDB Storage Engine......Page 174
Slow Query Basics: Optimize Data Access......Page 176
Are You Asking the Database for Data You Don’t Need?......Page 177
Rows examined and rows returned......Page 178
Rows examined and access types......Page 179
Complex Queries Versus Many Queries......Page 181
Chopping Up a Query......Page 182
Join Decomposition......Page 183
Query Execution Basics......Page 184
The MySQL Client/Server Protocol......Page 185
Query states......Page 187
The Query Optimization Process......Page 188
The query optimizer......Page 189
MySQL’s join execution strategy......Page 194
The execution plan......Page 196
The join optimizer......Page 197
Sort optimizations......Page 200
Returning Results to the Client......Page 202
Correlated Subqueries......Page 203
When a correlated subquery is good......Page 204
Index merge optimizations......Page 207
Equality propagation......Page 208
Loose index scans......Page 209
SELECT and UPDATE on the same table......Page 211
What COUNT() does......Page 212
Simple optimizations......Page 213
Optimizing JOIN Queries......Page 214
Optimizing GROUP BY and DISTINCT......Page 215
Optimizing LIMIT and OFFSET......Page 217
Optimizing UNION......Page 218
Query Optimizer Hints......Page 219
User-Defined Variables......Page 222
Be Careful with MySQL Upgrades......Page 227
The MySQL Query Cache......Page 228
How MySQL Checks for a Cache Hit......Page 229
How the Cache Uses Memory......Page 230
When the Query Cache Is Helpful......Page 233
How to Tune and Maintain the Query Cache......Page 235
Reducing fragmentation......Page 236
Improving query cache usage......Page 237
InnoDB and the Query Cache......Page 239
Alternatives to the Query Cache......Page 240
Storing Code Inside MySQL......Page 241
Stored Procedures and Functions......Page 243
Triggers......Page 244
Events......Page 246
Cursors......Page 248
Prepared Statements......Page 249
The SQL Interface to Prepared Statements......Page 251
Limitations of Prepared Statements......Page 253
User-Defined Functions......Page 254
Views......Page 255
Updatable Views......Page 257
Performance Implications of Views......Page 258
Limitations of Views......Page 260
Defaults for creating objects......Page 261
Settings for client/server communication......Page 262
Special-case behaviors......Page 263
Choosing a Character Set and Collation......Page 264
How Character Sets and Collations Affect Queries......Page 265
Full-Text Searching......Page 268
Natural-Language Full-Text Searches......Page 269
Boolean Full-Text Searches......Page 271
Full-Text Changes in MySQL 5.1 and Beyond......Page 272
Full-Text Tradeoffs and Workarounds......Page 273
Full-Text Tuning and Optimization......Page 275
Foreign Key Constraints......Page 276
Merge Tables and Partitioning......Page 277
Merge Tables......Page 278
Merge table performance impacts......Page 279
Merge table strengths......Page 280
Partitioned Tables......Page 281
Why partitioning works......Page 282
Partitioning examples......Page 283
Partitioned table limitations......Page 284
Optimizing queries against partitioned tables......Page 285
Distributed (XA) Transactions......Page 286
Internal XA Transactions......Page 287
External XA Transactions......Page 288
Optimizing Server Settings......Page 289
Configuration Basics......Page 290
Syntax, Scope, and Dynamism......Page 291
Side Effects of Setting Variables......Page 292
Getting Started......Page 294
General Tuning......Page 295
How much memory can MySQL use?......Page 296
Reserving memory for the operating system......Page 297
The MyISAM Key Cache......Page 298
The MyISAM key block size......Page 300
The InnoDB Buffer Pool......Page 301
The Thread Cache......Page 302
The Table Cache......Page 303
The InnoDB Data Dictionary......Page 304
MyISAM I/O Tuning......Page 305
InnoDB I/O Tuning......Page 307
The InnoDB transaction log......Page 308
How InnoDB opens and flushes log and data files......Page 312
The InnoDB tablespace......Page 314
The doublewrite buffer......Page 317
Other I/O tuning options......Page 318
MyISAM Concurrency Tuning......Page 319
InnoDB Concurrency Tuning......Page 320
Optimizing for BLOB and TEXT Workloads......Page 322
Optimizing for filesorts......Page 324
Inspecting MySQL Server Status Variables......Page 325
Tuning Per-Connection Settings......Page 328
Operating System and Hardware Optimization......Page 329
Which Is Better: Fast CPUs or Many CPUs?......Page 330
Scaling to Many CPUs and Cores......Page 332
Balancing Memory and Disk Resources......Page 333
Random Versus Sequential I/O......Page 334
Caching, Reads, and Writes......Page 335
What’s Your Working Set?......Page 336
The working set and the cache unit......Page 337
Finding an Effective Memory-to-Disk Ratio......Page 338
Choosing Hard Disks......Page 339
RAID Performance Optimization......Page 341
RAID Failure, Recovery, and Monitoring......Page 344
Balancing Hardware RAID and Software RAID......Page 345
The RAID stripe chunk size......Page 346
The RAID cache......Page 347
Storage Area Networks......Page 349
Using Multiple Disk Volumes......Page 350
Network Configuration......Page 352
Choosing an Operating System......Page 354
Choosing a Filesystem......Page 355
Swapping......Page 358
How to Read vmstat Output......Page 360
How to Read iostat Output......Page 362
A CPU-Bound Machine......Page 363
An I/O-Bound Machine......Page 364
An Idle Machine......Page 365
Replication Overview......Page 367
Problems Solved by Replication......Page 368
How Replication Works......Page 369
Creating Replication Accounts......Page 371
Configuring the Master and Slave......Page 372
Starting the Slave......Page 373
Initializing a Slave from Another Server......Page 376
Recommended Replication Configuration......Page 377
Statement-Based Replication......Page 379
Row-Based Replication......Page 380
Replication Files......Page 381
Sending Replication Events to Other Slaves......Page 383
Replication Filters......Page 384
Master and Multiple Slaves......Page 386
Master-Master in Active-Active Mode......Page 387
Master-Master in Active-Passive Mode......Page 389
Ring......Page 391
Master, Distribution Master, and Slaves......Page 393
Tree or Pyramid......Page 394
Selective replication......Page 395
Data archiving......Page 396
Emulating multimaster replication......Page 397
Creating a log server......Page 398
Replication and Capacity Planning......Page 400
Plan to Underutilize......Page 401
Monitoring Replication......Page 402
Measuring Slave Lag......Page 403
Determining Whether Slaves Are Consistent with the Master......Page 404
Resyncing a Slave from the Master......Page 405
Planned promotions......Page 406
Locating the desired log positions......Page 408
Switching Roles in a Master-Master Configuration......Page 411
Errors Caused by Data Corruption or Loss......Page 412
Mixing Transactional and Nontransactional Tables......Page 415
Data Changes on the Slave......Page 416
Undefined Server IDs......Page 417
Missing Temporary Tables......Page 418
Not Replicating All Updates......Page 419
Lock Contention Caused by InnoDB Locking Selects......Page 420
Writing to Both Masters in Master-Master Replication......Page 422
Excessive Replication Lag......Page 423
Don’t duplicate the expensive part of writes......Page 425
Prime the cache for the slave thread......Page 426
No Disk Space......Page 428
How Fast Is Replication?......Page 429
The Future of MySQL Replication......Page 431
Scaling and High Availability......Page 433
Terminology......Page 434
Planning for Scalability......Page 436
Buying Time Before Scaling......Page 437
Scaling Up......Page 438
Scaling Out......Page 439
Functional partitioning......Page 440
Data sharding......Page 441
Choosing a partitioning key......Page 443
Querying across shards......Page 445
Allocating data, shards, and nodes......Page 446
Fixed allocation......Page 448
Dynamic allocation......Page 450
Explicit allocation......Page 451
Rebalancing shards......Page 452
Generating globally unique IDs......Page 453
Tools for sharding......Page 455
Scaling Back......Page 456
Keeping active data separate......Page 458
Clustering......Page 459
Load Balancing......Page 460
Splitting reads and writes in replication......Page 462
Changing the application configuration......Page 464
Moving IP addresses......Page 465
Load balancers......Page 466
Load-balancing algorithms......Page 468
Load Balancing with a Master and Multiple Slaves......Page 469
Planning for High Availability......Page 471
Replicated-disk architectures......Page 473
Synchronous MySQL replication......Page 475
Failover and Failback......Page 476
Virtual IP addresses or IP takeover......Page 477
Middleman solutions......Page 478
Handling failover in the application......Page 479
Find the Source of the Problem......Page 481
Look for Common Problems......Page 482
Web Server Issues......Page 484
Finding the Optimal Concurrency......Page 486
Caching......Page 487
Caching Below the Application......Page 488
Application-Level Caching......Page 489
Cache Control Policies......Page 491
Cache Object Hierarchies......Page 492
Pregenerating Content......Page 493
Extending MySQL......Page 494
Alternatives to MySQL......Page 495
Backup and Recovery......Page 496
It’s All About Recovery......Page 497
The Big Picture......Page 499
Why Backups?......Page 500
What Can You Afford to Lose?......Page 501
Online or Offline Backups?......Page 502
Logical backups......Page 503
Raw backups......Page 504
What to Back Up......Page 505
Incremental backups......Page 506
Storage Engines and Consistency......Page 507
File consistency......Page 508
Replication......Page 509
Managing and Backing Up Binary Logs......Page 510
The Binary Log Format......Page 511
Backing Up Data......Page 512
SQL dumps......Page 513
Delimited file backups......Page 514
Parallel dump and restore......Page 515
How LVM snapshots work......Page 516
Prerequisites and configuration......Page 517
Creating, mounting, and removing an LVM snapshot......Page 518
Lock-free InnoDB backups with LVM snapshots......Page 520
Planning for LVM backups......Page 522
Recovering from a Backup......Page 523
Restoring Raw Files......Page 524
Starting MySQL after restoring raw files......Page 525
Loading SQL files......Page 526
Loading delimited files......Page 527
Point-in-Time Recovery......Page 528
Recovering with a log server......Page 530
InnoDB Recovery......Page 531
Causes of InnoDB corruption......Page 532
How to recover corrupted InnoDB data......Page 533
Backup and Recovery Speed......Page 534
mysqldump......Page 535
InnoDB Hot Backup......Page 537
mk-parallel-dump......Page 538
Installing and testing ZRM......Page 539
Comparison of Backup Tools......Page 541
Scripting Backups......Page 542
Terminology......Page 545
Privileges......Page 546
The Grant Tables......Page 547
How MySQL Checks Privileges......Page 548
Adding, Removing, and Viewing Grants......Page 549
Setting Up MySQL Privileges......Page 550
Privilege Changes in MySQL 5.0......Page 553
Triggers......Page 554
Views......Page 555
Privileges and Performance......Page 556
Connecting through localhost versus 127.0.0.1......Page 557
Disabling anonymous users......Page 558
Don’t grant privileges on the mysql database......Page 559
Granting privileges on wildcarded databases......Page 560
Invisible privileges......Page 561
Obsolete privileges......Page 564
Guidelines......Page 565
Network Security......Page 566
Localhost-Only Connections......Page 567
No default route......Page 568
Connection Encryption and Tunneling......Page 569
SSL in MySQL......Page 570
TCP Wrappers......Page 572
Automatic Host Blocking......Page 573
Hashing Passwords......Page 574
Encrypted Filesystems......Page 575
Design issues......Page 576
MySQL in a chrooted Environment......Page 578
System Variables......Page 581
SHOW STATUS......Page 582
Binary Logging Status......Page 583
Handler Operations......Page 584
SELECT Types......Page 585
Sorts......Page 586
InnoDB-Specific......Page 587
Miscellaneous......Page 588
Header......Page 589
SEMAPHORES......Page 590
LATEST FOREIGN KEY ERROR......Page 591
LATEST DETECTED DEADLOCK......Page 593
TRANSACTIONS......Page 596
FILE I/O......Page 598
INSERT BUFFER AND ADAPTIVE HASH INDEX......Page 599
BUFFER POOL AND MEMORY......Page 600
ROW OPERATIONS......Page 601
SHOW PROCESSLIST......Page 602
SHOW MUTEX STATUS......Page 603
Replication Status......Page 604
INFORMATION_SCHEMA......Page 605
MySQL Visual Tools......Page 607
SQLyog......Page 608
Monitoring Tools......Page 609
Homegrown systems......Page 610
Alternatives to Nagios......Page 611
MySQL Monitoring and Advisory Service......Page 612
RRDTool-based systems......Page 613
innotop......Page 615
HackMySQL Tools......Page 619
Maatkit Analysis Tools......Page 620
MySQL Proxy......Page 622
Maatkit Utilities......Page 623
Sources of Further Information......Page 625
Copying Files......Page 627
A One-Step Method......Page 628
Other Options......Page 629
File Copy Benchmarks......Page 630
Invoking EXPLAIN......Page 631
Rewriting Non-SELECT Queries......Page 633
The id Column......Page 634
The table Column......Page 636
An example of complex SELECT types......Page 637
The type Column......Page 639
The key_len Column......Page 641
The ref Column......Page 642
The rows Column......Page 643
The filtered Column......Page 644
Visual EXPLAIN......Page 645
Overview: A Typical Sphinx Search......Page 647
Efficient and Scalable Full-Text Searching......Page 651
Applying WHERE Clauses Efficiently......Page 652
Finding the Top Results in Order......Page 653
Optimizing GROUP BY Queries......Page 654
Generating Parallel Result Sets......Page 655
Scaling......Page 656
Aggregating Sharded Data......Page 658
Architectural Overview......Page 659
Typical Partition Use......Page 660
Special Features......Page 661
Phrase Proximity Ranking......Page 662
Filtering......Page 663
The SphinxSE Pluggable Storage Engine......Page 664
Advanced Performance Control......Page 665
Full-Text Searching on Mininova.org......Page 667
Full-Text Searching on BoardReader.com......Page 668
Optimizing Selects on Sahibinden.com......Page 669
Optimizing GROUP BY on BoardReader.com......Page 671
Optimizing Sharded JOIN Queries on Grouply.com......Page 672
Conclusion......Page 673
Lock Waits at the Server Level......Page 674
Table Locks......Page 675
Finding out who holds a lock......Page 677
Name Locks......Page 678
User Locks......Page 679
InnoDB Lock Waits......Page 680
Toward more usable lock output......Page 681
Falcon Lock Waits......Page 682
Index......Page 683