Get content from almost any site to your WordPress blog, automatically!
FOR WHAT IT CAN BE USED
- Create a personal site which collects news, posts, etc. from your favorite sites to see them in one place
- Use it with WooCommerce to collect products from shopping sites
- Collect products from affiliate programs to make money
- Collect posts to create a test environment for your plugin/theme
- Collect plugins, themes, apps, images from other sites to create a collection of them
- Keep track of competitors
- You can imagine anything. The internet is full of contents
HOW IT WORKS
It’s all about CSS selectors and you can learn how to use them in minutes by watching the introduction tutorial. The plugin’s Visual Inspector tool also helps you find CSS selectors easily by clicking onto the elements in the target sites. Here is the gist of it:
WHAT WP CONTENT CRAWLER CAN DO
Here is the list of some features of WP Content Crawler. To learn about all of the features, please see the features table below.
WHY WP CONTENT CRAWLER
Problems with crawling a website
- Not an easy task, requires advanced programming skills
- Every website is different and needs tailored crawling implementation
- Not just every website is different, but also pages of a single website can differ
- Pages and their source codes need to be investigated intensively to come up with a crawling plan
- Knowing how to save certain information in a specific place in WordPress requires knowledge about the internal structure of WordPress and how WordPress works
- If certain information should be saved into a specific field defined by a third-party plugin, one should modify the crawling implementation after researching for hours about how to save that information
- One should know about how HTML works and how to extract certain parts from HTML code
- One should handle all possible inconsistencies that might be in the source codes of websites to provide a robust solution that will keep working
- What if the posts need to be shared in regular time intervals?
- What if you want to crawl new posts added to a website after some time?
- What about translating the posts from one language to another?
- What if the posts needs to be paraphrased to provide a better search engine optimization for the website?
- What if some information should not be retrieved?
- What if certain information should be changed to make it suitable for your site?
- What if another site needs to be crawled, not just one?
- What if that other site needs a different crawling plan?
- What if you need to login to the website to crawl it?
- What if the website changes its source code?
- What if you want to update the crawled posts by recrawling them from the original website?
- What if you want to make sure if the information is retrieved exactly as you want it before automatically posting the posts to your website?
- What if you want to ensure your site’s security by making sure no malicious-code-executing code ends up in your site?
- And many more what-ifs that you might not even imagine unless you come across them
Our vision and mission
We believe that robust, reliable, and automated crawling capabilities should be available for anyone. We want to democratize this field by letting anyone have these capabilities, not just developers. With this purpose, we aim to provide a plugin that you will fall in love with and feel at home when using it. To let it accessible by anyone, we make the plugin low-cost and easy-to-use. We do not implement the features just to make sales. We plan and execute for the future. We always listen to your feedback and make required changes accordingly. We think that WordPress plugins should be developed with enterprise-level care. So, we intensively test the plugin before each release with automated end-to-end UI tests, currently over 1700 tests, that run in many different environments in the cloud for a total of over 40 hours to ensure the plugin is compatible with your server and WordPress environments and you, our valuable customers, get the quality and reliability you deserve.
How we solve these problems
We have been developing WP Content Crawler for almost 4 years such that we have come across almost all the what-ifs. Working with our customers and listening to their needs, we provide robust and reliable solutions to these problems. We believe that one should just provide from which site the information should be retrieved and what information should be retrieved from that website and then start crawling that site, without worrying about the complex behind-the-scenes operations.
To make it available to anyone, we provide a detailed online documentation that contains not just the description of the settings but how to use the settings to achieve your goals. Sometimes you might not feel like reading the documentation. We also provide interactive step-by-step guides that are available in the plugin, just one click away. You can start the interactive guides showing you step-by-step how you can do certain things any time and from any step you want.
One of the most distinctive features of WP Content Crawler is the ability to test almost any configuration. By this way, you will not come across any surprises after you enable automatic crawling. When testing, the errors related to your settings are shown so that you can fix them before they cause any problems.
WP Content Crawler has so many features that even we do not know how many of them are there. You can automatically crawl, update, and delete the posts, you can translate posts, spin posts, you can even define what fields need to be translated or spun if you do not want them all changed. You can find-replace almost anything. You can assign some information from the target post to a short code and place that information anywhere in the post. You can save WooCommerce products. You can save details for third-party plugins that we do not even know they exist. The features of the plugin are designed such that you feel that you are in control when you use them. We make them as flexible as possible to make them fit your needs. When designing new features, we always keep in mind that you might need a more advanced version of that feature and we design the features accordingly. We ensure that the features and the entire code of the plugin are maintainable and extendable so that we can always improve the plugin.