Magento Expert Forum - Improve your Magento experience

Results 1 to 14 of 14

Import faster in Magento

  1. #1
    Junior Member golddev's Avatar
    Join Date
    Mar 2013
    Posts
    45
    Thanks
    1
    Thanked 6 Times in 2 Posts

    Cool Import faster in Magento

    A lot of import scripts Ive worked with previously are very slow. Its true to say that Magento is a bit of a memory and process hog, but there are a few things that we can do to speed up import processes. Its worth noting that these dont apply if youre doing direct database updating; they only apply if you use the API/Dataflow.

    It is very important to have a good specification of server to run Magento in the first place, and imports are likely to push the server to its limits. Instead of a transactional lifestyle, the server is going to be hit repeatedly until the data you want to import has been added.


    I wouldnt recommend doing any sort of import onto a live box unless you absolutely must. Simply because it will impact on the performance of your main site for a sustained period. You can of course run imports as scheduled tasks out of normal operating hours, or importing into a staging version of your site and then merge across at a time of convenience which can be quite tricky.

    There are a few things you can do to improve the speed of imports, and to make sure that your imports dont fail.

    Memory limit



    Set the the memory limit for your script so that it has enough to get the job done, and not too much that it will impact on your site. Simply add this at the top of your script, setting the size to be something appropriate:

    Code:
    ini_set('memory_limit', '1024M');
    This all depends on how much memory is in your box, how many products/orders etc youre working on with your script.

    Magento Store



    Set the current store for Magento before beginning your import. In this case, we only have one store ID in our Magento setup, so we can just set it to the admin store id.

    Code:
    Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

    Indexing



    Magento has a very thorough caching and indexing system. Every time you add or edit a product, or amend an attribute, the caching system updates and also re-indexes the data. This improves front-end load times and accuracy of search data as well as many other things. This can get in the way when were doing an import though Magento would have to do this every time a product is added/updated and we might be adding or updating thousands of records during an import. Its safe to say that this is probably the case in any running store. Youve probably already noticed that Magento has the ability to set these to manual.

    Were going to leverage this so that our script runs quicker. The theory is that we can suspend indexing, complete our process, and then re-enable & re-run indexing. We can therefore end up with two functions unSetIndex() which we can call at the beginning of our process, and setIndex() that we can call at the end to change things back. Obviously, you only need to call these in your script as and when you want, i.e. if you dont have indexes set on staging, dont call them.

    PHP Code:
    private function unSetIndex()
    {
        
    $processes Mage::getSingleton('index/indexer')->getProcessesCollection();
        
    $processes->walk('setMode', array(Mage_Index_Model_Process::MODE_MANUAL));
        
    $processes->walk('save');

        return 
    true;
    }

    private function 
    setIndex()
    {
        
    $processes Mage::getSingleton('index/indexer')->getProcessesCollection();
        
    $processes->walk('reindexAll');
        
    $processes->walk('setMode', array(Mage_Index_Model_Process::MODE_REAL_TIME));
        
    $processes->walk('save');

        return 
    true;

    As you can see in the setIndex() function, we make a call to reindexAll. You could omit this is you didnt actually want to trigger the re-index and had it scheduled. It would probably be a bit mean to make the next person to save a product wait while it re-indexes everything.

    Time your scripts



    This tip doesnt directly speed up your script, but its good to benchmark your script as you put it together and do test runs. This will also give you some insight into where youre at when comparing your development, staging and live environments.

    Ive had this in my arsenal for quite some time, so dug out a reference to it http://www.developerfusion.com/code/...n-time-in-php/

    PHP Code:
    <!-- put this at the top of the page -->
    <!--?
    php 
       $mtime 
    microtime();
       
    $mtime explode(" ",$mtime);
       
    $mtime $mtime[1] + $mtime[0];
       
    $starttime $mtime;
    ?-->

    <!-- 
    put other code and html in here -->

    <!-- 
    put this code at the bottom of the page -->
    <!--?
    php 
       $mtime 
    microtime();
       
    $mtime explode(" ",$mtime);
       
    $mtime $mtime[1] + $mtime[0];
       
    $endtime $mtime;
       
    $totaltime = ($endtime $starttime);
       echo 
    "This page was created in ".$totaltime." seconds";
    ?--> 
    Hopefully, youve got a few tips to make your import much quicker now! Any problems leave them in the comments below.

    Source: branded3.com

    View more threads in the same category:


  2. #2
    New member
    Join Date
    Aug 2014
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by golddev View Post
    It is very important to have a good specification of server to run Magento in the first place, and imports are likely to push the server to its limits. Instead of a transactional lifestyle, the server is going to be hit repeatedly until the data you want to import has been added.
    Hi,

    thanks for your contribution. Just interested what would you consider good enough server specification?
    __________________________________
    Custom software development company in New York

  3. #3
    Administrator david's Avatar
    Join Date
    Nov 2012
    Posts
    260
    Thanks
    23
    Thanked 40 Times in 33 Posts

    Default

    Quote Originally Posted by stakejake View Post
    Hi,

    thanks for your contribution. Just interested what would you consider good enough server specification?
    You can check Magento system requirements

  4. #4
    Junior Member Magento MageSolution's Avatar
    Join Date
    Dec 2013
    Posts
    203
    Thanks
    0
    Thanked 5 Times in 5 Posts

    Default

    You may look at my tutorial with some more task to improve your site speed :
    Take it easy by following these steps to Speed up Magento. We are pretty sure you will be happy with the result.

    1.Copy HTTP Request by Expires Headers
    2.Reduce the file size of images to be uploaded
    3.Turn off unnecessary modules
    4.Utilize Magento Compilation
    5.Enable Fat Catalog in Admin panel
    6.Enable Merge CSS and JS
    7.Clean Database Log
    For Details : How to optimize Magento site

  5. #5
    Junior Member
    Join Date
    Aug 2014
    Posts
    24
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Our Magneto optimization lets your customers benefit from faster page loads, promoting longer time on site and increased sales conversions.

  6. #6
    Junior Member michealnguyen's Avatar
    Join Date
    Sep 2014
    Posts
    97
    Thanks
    2
    Thanked 8 Times in 8 Posts

    Default

    Here are some tips:
    - Disable auto index on save
    - Using direct database Query instead of Model if you are strong at Magento ^^

  7. #7
    New member
    Join Date
    Sep 2014
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Magento development for the best way to the ecommerce

    Current day web designers are best in sense they search out Magento like web design stage for ecommerce solutions. Magento offers website design with comfortable and in simple way.

  8. #8
    Junior Member michealnguyen's Avatar
    Join Date
    Sep 2014
    Posts
    97
    Thanks
    2
    Thanked 8 Times in 8 Posts

    Default

    You should change Re-index mode to Manually instead of On save.

    If you want advanced Import Export products try to use http://magegiant.com/magento-import-export-products/

  9. #9
    Junior Member
    Join Date
    Oct 2014
    Posts
    10
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Thanks Golddev, some useful advice for importing into Magento

  10. #10
    New member
    Join Date
    Jan 2015
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Thank you very much for your valuable input.store information sharing.God bless you70-461

  11. #11
    New member itszuma's Avatar
    Join Date
    Feb 2015
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Looking for advice on doing a bulk import of 2000+ products to my magento CE store. The store already has over 1000 products loaded. Looking to add several thousands more products from 2-4 different vendor companies.
    Q. Do the .csv files that I would use for the import, need to have the exact same column heading names as the database that the site is already using for the products already loaded? PS.. the .csv file includes images, at least one per product.

    Or put another way, can the column heading names for the .csv files from the new vendor companies, be different from the column headings being used by the product database that the site is currently using..?? Thank you to anyone that can help educate me on this.

  12. #12
    New member
    Join Date
    Dec 2016
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Our full-service digital agency will get the job done for you. App development, video production, online marketing, SEO, social media and everything you need to raise your sales. Just contact for solved your importance issue Digital Marketing Solution Miami

  13. #13
    Junior Member
    Join Date
    Dec 2016
    Posts
    78
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Good sharing, man

  14. #14
    Junior Member James's Avatar
    Join Date
    Jan 2016
    Posts
    629
    Thanks
    2
    Thanked 13 Times in 12 Posts

    Default

    Thanks for sharing!
    What's about import and exporting product views in Magento 2? Beside using extension like that, is there any other solution?

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •