Frequently Asked Questions

What’s a “WP Hive Plugin Test”?

WP Hive tests a plugin in its very basic form. All WP Hive tests are automated and blind. The concept of the blind test was first introduced in 1784 by the French Academy of Sciences. You can learn more about it on Wikipedia.

• Whether the plugin can be activated or not.
• Whether it throws some errors (JS error/resource errors/PHP error/activation error etc)
• How much memory a plugin takes after activating it.
• How much impact a plugin has on the server.
• Whether the database footprint of the plugin is optimal or not
• How frequently the plugin gets updated
• Whether the plugin is compatible with the latest WordPress and PHP versions.

What’s the Test Environment?

All the plugins are tested on the same server with exactly the same configuration. The server configuration is as follows

• The test sites are hosted on Google Cloud VM instances, with one site/plugin per instance. The machine type is n1-standard-1. The server is an 8-core CPU with 8GB of RAM.
• The test sites are hosted on the Apache/2 server.
• All tests are done on PHP 7.2.16
• The database server is MySQL 8.0.15
• The default PHP memory limit is 256MB.

How WP Hive checks WordPress compatibility?

Our WordPress compatibility check combines a few conditions. If all conditions are met, we show “Fully Compatible.” The conditions are:

• No PHP Errors
• No JS Errors
• No resource Errors

How WP Hive handles each plugin’s data? How are you generating the benchmark results?

WP Hive has multiple parts. One of them is a very powerful crawler. The crawler crawls all the plugin from WordPress.org, blindly activates it, and logs many crucial data such as page speed, memory usage, JavaScript issues, activations issues and more. It is always looking an eye on the WordPress.org repository.

• All the scripts run on a VPS with 8 CPU cores and 8 GB of RAM. This server is almost certainly more powerful than it needs to be for this project.
• The test sites are hosted on Google Cloud VM instances, one site/plugin per instance. The machine type is n1-standard-1. Whenever the crawler finds there is a new plugin (or plugin update) that hasn’t been tested before – the testing script tests the plugin, records the new data, sends the data back.
• The frontend gets updates with the updated test results with new plugin release. Yes, the whole process is automatic.
• Whenever there is a new plugin update – the crawler gets it within 1-3 days, and the process repeats.

How WP Hive declares whether a plugin has minimal impact on memory usage or not?

While setting a standard on memory friendliness, we got into some trouble. Because there hasn’t been any standard in the WordPress industry to choose memory friendliness.

So we thought, we don’t need to know whether it is 100KB or 5MBs for each plugin. Let the WordPress industry decide that. So, we dynamically calculate the average memory usage of 50000+ WordPress plugins (all plugin data we have).

• First, we calculate an average of the memory usage of all the plugin data we have. This value is generated dynamically and changes each day.
  
  Then we add 200KB with that value as C. Let’s call the new value M. WP Hive takes the decision to call a plugin memory friendly or not by taking into account the value of M.

Here, we are assuming the value C is constant and is 200KB. This value is added to provide legitimacy to our results. For example, we do not want to declare every plugin as “Not memory friendly” since the average value can vary each day we are adding new plugins every day to our database. By adding 200KB to the equation as C – we are giving every plugin a fair chance to gain the memory friendly status on WP Hive.

• If the plugin’s memory usage is less than M – WP Hive declares it as memory friendly.

• If the plugin’s memory usage is greater than M, WP Hive declares it as not memory friendly.

How does WP Hive declare whether a plugin has a minimal impact on pagespeed or not?

While setting a standard on the pagespeed, we followed the same principle as before.

We dynamically calculate the average page speed of 50000+ WordPress plugins (all plugin data we have).

• First, we calculate an average of the pagespeed of all the plugin data we have. This value is generated dynamically and changes each day.
  
  Then we add 1000 milliseconds with that value. Let’s call the new value M. WP Hive takes the decision to call a plugin’s pagespeed optimal or not by taking into account the value of M.

Here, we are assuming the value C is constant and is 1000 milliseconds. This value is added to provide legitimacy to our results. For example, we do not want to declare every plugin as “not pagespeed friendly” since the average value can vary each day we are adding new plugins every day to our database. By adding 1000 milliseconds to the equation as C – we are giving every plugin a fair chance to gain the pagespeed-friendly status on WP Hive.

• If the plugin’s page speed score is less than M – WP Hive declares that the plugin has “minimal impact on page speed”
• If the plugin’s page speed score is greater than M – WP Hive declares that the plugin does not qualify for the “minimal impact on pagespeed” status.

How does WP Hive find out whether a plugin has PHP/JS/Resource/Resource Error?

All WP Hive tests are automated and blind. The results are not manipulated in any way. While doing a “WP Hive Plugin Test” we test each plugin blindly.

The system automatically activates a plugin and looks if the plugin generates any error. The whole process is automatic.

WP Hive shows the actual error (if there is any while activating the plugin) and shows that in the Errors section.

How does WP Hive understand whether the plugin is frequently updated or not?

We figure out whether the plugin has been updated in the last 90 days or not. If the plugin is updated in the last 90 days – we show the status the plugin is frequently updated.

Which plugins are tested?

WP Hive aims to test EVERY WORDPRESS PLUGIN out there. We know it’s a hard task but we already have 50000 plugin data and we are updating the data as soon as there is a new release for the plugin. I guess we are already covered!

We’ve tested about 98% of WordPress plugins from the WordPress repository at least once. However, due to some technical constraints – we cannot test all the plugins. For example, if a plugin doesn’t have a “version” header or our test script fails to activate the plugin – we cannot test them.

Does a positive result on WP Hive mean that the plugin is guaranteed to work?

Not really. This is an automated test and the test criteria are basic. There might be more bugs that WP Hive cannot find out. Also, all plugins are tested in the same environment (WordPress version + PHP version + server settings).

If you use a different server, different PHP versions, and different configurations – you still might run into issues and the results might not be correct for you. WP Hive gives an overview of a plugin, not a final judgment.

If a plugin doesn’t pass certain WP Hive criteria, does it mean the plugin is broken and unusable?

NO. It only suggests that there might be something wrong with the plugin. It hasn’t passed our criteria. However, it doesn’t necessarily mean that the plugin is unusable.

• WP Hive cannot distinguish between critical bugs and harmless errors. For example, we treat every database error or resource error as a major problem. In practice, the issues may have no visible effect on your site.
  
• WP Hive doesn’t care about dependency. For example, a particular plugin might clearly say that you must install another plugin first to use the plugin. However, WP Hive ignores all of that. WP Hive installs each plugin on a fresh WordPress site and activates it using the default settings. If the plugin crashes, the test result will identify as an error.
  
• All tests are done with a headless browser. There might be a few plugins that trigger errors but the plugins may work perfectly fine in a “real” browser. So, there might be false positives. However, the case is extremely unlikely.