activerecord-import gets basic SQLite3 and PostgreSQL support

on April 09, 2010 @ 03:44 AM

A few weeks back I started working on extracting the import functionality from ar-extensions into its own library. The initial extraction involved pulling out all of the MySQL functionality first, test by test, piece by piece. Well, tonight I pulled in the basic support for SQLite3 and PostgreSQL.

While it’s working great for MySQL, SQLite3, and PostgreSQL it has not been released as its own rubygem yet. I expect that to happen in the near-term future. In the meantime you can always check out the project on github:

• http://github.com/zdennis/activerecord-import/

In the past folks have emailed about sparse documentation. When the gem is released documentation will be provided by the github wiki ( http://wiki.github.com/zdennis/activerecord-import/ ) and the most up-to-date will be provided in the RDOC itself.

I’ve ran some benchmarks recently against MySQL 5.1 and you can find the results here: http://wiki.github.com/zdennis/activerecord-import/benchmarks

More to come!

ActiveRecord Import Updates

on March 12, 2010 @ 05:17 AM

Nearly four years ago I embarked on figuring out ActiveRecord internals enough so I could have a library which allowed me to perform bulk inserts of data in an optimized way based on the database adapter being used. Over those four years, I’ve maintained and updated the functionality with contributions from the community. A big thanks to Blythe Dunham, James Herdman, Gabe da Silveira, Thibaud Guillaume-Gentil, Marcus Crafter, and Mark Van Holstyn.

Now that Rails is nearing a 3.0 release, it’s time for to revisit this import functionality, make it compatible with the heavily refactored and cleaned up ActiveRecord internals (thx Rails core team) and add some more polished features and implementation.

The import functionality which was in ActiveRecord::Extensions is being extracted into its own library, activerecord-import. This library will be smaller and its only purpose will be to provide mass import functionality.

The work has already begun over on github: http://github.com/zdennis/activerecord-import

I look forwarding to officially releasing this in the upcoming weeks!

ar-extensions 0.8.2 released

on March 17, 2009 @ 02:09 AM

ar-extensions 0.8.2 is released. This release fixes compatibility issues with Rails 2.3.1 and Rails 2.3.2. It should show up soon at a rubygems mirror near you.

Thanks to Stephen Heuer, Glenn Rempe, and Michael Murray.

gem install ar-extensions -v 0.8.2

ar-extensions 0.8.1 released

on February 10, 2009 @ 01:15 AM

ar-extensions 0.8.1 is released. This is primarily a bug fix release to ensure it works successfully with Rails 2.2.2. Here are the changes:

• fixed issue in http://zdennis.lighthouseapp.com/projects/14379/tickets/14-join-table-conditions-broken
• Updated usage of Inflector to point to ActiveSupport::Inflector
• Fixed bug where finder conditions which use arext-suffixed keys and normal field keys would fail when using a conditions hash. (Gabe da Silveira)
• added timestamps options to not automatically add timestamps even if record timestamps is disabled in ActiveRecord::Base (Thibaud Guillaume-Gentil)
• Updated to use alias_method_chain so that multiple aliased finders remain intact (Marcus Crafter)

It may take a little while for the gem to show up in the rubygem mirrors. Thanks to everyone who contributed!

ar-extensions 0.8.0, forgot to mention

on August 16, 2008 @ 02:17 PM

I forgot to mention that ar-extensions no longer loads adapter specific functionality by itself. You need to tell it what you want. For example if you want to load import functionality for MySQL you’d have to require the right files, like so:

1
2

require 'ar-extensions/adapters/mysql'
require 'ar-extensions/import/mysql'

Sorry about that.

ar-extensions 0.8.0 released!

on August 06, 2008 @ 07:46 PM

ar-extensions 0.8.0 is finally released! All known bugs and compatibility issues with Rails 2.1.0 have been resolved.

• 0.8.0 does not work with the latest Rails edge due to some additional features on edge that weren’t in 2.1.0
• 0.8.0 should not be used with Rails 2.0.2 or below

If any feature requests or bugs please report them on lighthouse

Installing

gem install ar-extensions

Oracle and PostgreSQL users

If you’re using ar-extensions 0.7.0 with Oracle or PostgreSQL I am very interested to find out if 0.8.0 works fine. I don’t have Oracle, and I’m not a PostgreSQL user. If you’re interested in running the test suite against one of those databases please let me know.

Thanks for your patience and enjoy!

ar-extensions, Rails 2.x followup

on July 18, 2008 @ 01:41 AM

First things first, ar-extensions moved to Lighthouse and Github. So please no more tickets or patches on rubyforge. I’ll close out or move over the ones on rubyforge that currently reside there.

• ar-extensions at Lighthouse
• ar-extensions at GitHub

Now, the fun stuff.

ar-extensions and Rails 2.x compability

For the most part ar-extensions 0.7.0 (and below) is not compatible with Rails 2.0.x and 2.1. The import functionality works, but ar-extensions will bork on some edge cases of using ActiveRecord::Base.find methods. It won’t return the wrong value, it will simply raise an exception. So if you’re using ar-extensions in your Rails 2.x app, the app itself will probably bork (and I realize that you probably already know this). If you’re using the import functionality in isolation then you should be fine.

Some of the loading issues with ar-extensions that people had are due to ar-extensions being loaded during the time that Rails goes through the loading process (if you put require ‘ar-extensions’ in config/environments/development.rb, test.rb or production.rb). You can get around this by moving if after the Rails::Initializer.run block in your config/environment.rb file.

Tonight I made several commits to get ar-extensions closer to Rails 2.1.0 compatibility. HEAD right now is Rails 2.0.1 compatible, but not 2.0.2 or 2.1.0 compatible. As soon as ar-extensions is compatible I will release 0.8.0. I am hoping for it to be released within the next few days. And by compatible I mean that ar-extensions not only passes its own tests, but it also doesn’t break any of ActiveRecord’s tests.

ar-extensions, Rails 2.x

on July 17, 2008 @ 02:34 AM

There are some issues with ar-extensions and Rails 2.x. At first glance it mainly has to do with load order and what to load. I’m going to try to post an update or at least instructions tomorrow so people can use ar-extensions with Rails 2.x and up.

I’m going to be moving the codebase to github and will start work on ar-extensions to get closer to a 1.0 release later this fall. I plan to kick out 0.8 which includes some bug fixes and any Rails 2.x issues as soon as possible.

I am also looking into using Lighthouse for tickets. Rubyforge’s bug tracking isn’t the greatest and apparently there have been tickets there for a while that I had no idea about (sorry about that).

0.9 will be the first release where I look to improve and cleanup the codebase. Thanks for your patience.

Any questions, comments or suggestions feel free to let me know. Thanks,

ar-extensions get a little praise

on January 20, 2008 @ 09:52 PM

Alan Miles found the import functionality of my ar-extensions gem/plugin helpful and he posted about it here

I’m so glad to hear that ar-extensions is helping Alan and it’s nice to see that he’s sharing this usefulness with others via his blog

ActiveRecord::Extensions 0.7.0 Released!

on July 21, 2007 @ 06:37 AM

ActiveRecord::Extensions 0.7.0 is released! Thanks to Michael Flester and Gabe da Silveira for the patches they submitted!

Updates include:

• Oracle support for better finders (but regexps only work with Oracle 10 or higher) and import (Michael Flester)
• created_at/updated_at timestamps are now implemented with import functionality if they exist on the tables (Zach Dennis/Michael Flester)
• fixed padding issue with MySQL by increasing the byte buffer size to 8 bytes. (thanks to Gabe da Silveira)

To install as a gem you can run:

gem install ar-extensions

To update a previously installed gem you can run:

gem update ar-extensions

To install as a script/plugin you can run:

script/plugin install http://arext.rubyforge.org/svn/tags/ar-extensions-0.7.0

To see information on the usage of ar-extensions please refer to the arext page or refer to the RDOC documentation

Enjoy!

For questions, comments please feel free to email me, zach dot dennis at gmail dot com or by submitting to rubyforge project page for arext.

Rails patch 8277

on May 06, 2007 @ 05:11 AM

The synchronize functionality released in ActiveRecord::Extensions 0.6.0 should probably go in core. So I submitted a patch

My reasoning is that is ActiveRecord::Base#reload already works on a per instance level, why not allow it work on a class level where the class can efficiently handle reloads on multiple instances at once.

The benefit of the class level reload is that you can do one query to the database regardless of how many instances you have rather then having to perform one query per instance.

If you’re in favor of the patch please encourage it gets accepted.

ActiveRecord::Extensions 0.6.0 released

on May 06, 2007 @ 04:34 AM

ActiveRecord::Extensions 0.6.0 is released! You can install it via RubyGems:

gem install ar-extensions

What’s New?

• support for arrays when using the Hash arguments and the ‘_contains’ suffix.
• a synchronize method which acts like ActiveRecord::Base#reload, but it works on multiple instances. The benefit being that it makes one query to reload all instances rather then one query per instance
• a bug fix for URI encoded strings when using find with ActiveRecord objects

Examples

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# synchronize example
post = BlogPost.find_by_author_name( 'zdennis' )
columns = [ :author_name, :title ]
values = [ [ 'yoda', 'test post' ] ]
BlogPost.import columns, values, :synchronize=>[ post ]
post.author_name # => 'yoda'

# using synchronize option with import
posts = BlogPost.find(:all)
BlogPost.import( columns, values, :synchronize=>posts)

# using arrays with the _contains suffix
BlogPost.find( 
  :all,  
  :conditions=>{ :author_name_contains=>[ 'ab', 'oc', 'ed' ] } )

Questions, Comments

zach dot dennis at gmail dot com

ActiveRecord::Extensions 0.5.2 Released!

on March 15, 2007 @ 01:32 AM

ActiveRecord::Extensions 0.5.2 is released! A rubygem has been born!

Installation is so easy now for ActiveRecord::Extensions:

gem install -r 'ar-extensions'

Enjoy!

0.5.1 was also released, but I quickly released 0.5.2 after it to fix a possible bug with method aliasing. This would only occur if you had another plugin or rubygem which used the same alias name for ActiveRecord::Base#quote

ActiveRecord::Extensions 0.5.0 Released!

on March 14, 2007 @ 02:14 AM

ActiveRecord::Extensions 0.5.0 is released! This release includes support for querying against Time objects against ActiveRecord columns which are considered :datetime by ActiveRecord’s standards .

Code Example

To find all blog posts within the last week:

BlogPost.find( :all, :conditions=>{ :created_at_gt => Time.now-1.week } )

To find all posts within the last year:

BlogPost.find( :all, :conditions=>{ :created_at_gt => Time.now-1.year } )

To find all posts over a year old:

BlogPost.find( :all, :conditions=>{ :created_at_lt => Time.now-1.year} )

User.find( :all, :conditions=>{ :logged_in_at_gt => Time.now-15.minutes} )

Supported Suffix Modifiers

• _lt => less than
• _gt => greater than
• _lte => less than or equal to
• _gte => greater than or equal to
• _ne => not equal to
• _not => not equal to
• eq => equal to _(this is implied without a suffix present)

MySQL TIMESTAMP Disclaimer

This does not work on TIMESTAMP columns for MySQL due to an issue with the inconsistency of TIMESTAMP columns in varying versions of MySQL and due to the nature of server configurations which can alter the behavior of TIMESTAMP columns.

PostgreSQL Support Note

I recently switched to a Mac, I am compiling PostgreSQL 8.1 and 8.2 as I type this. As soon as I can test this functionality against PostgreSQL I will release an update with that support.

Download It Now!

• Rubyforge: http://rubyforge.org/frs/?group_id=2113

ActiveRecord::Extensions 0.4.0 Released!

on February 12, 2007 @ 02:22 AM

ActiveRecord::Extensions 0.4.0 is released!

Updates

• added to_csv functionality (works with belongs_to, has_one and has_many relationships)
• added temporary table functionality
• added foreign key functionality

to_csv functionality

Example 1, exporting all fields

1
2
3
4

 class Book < ActiveRecord::Base ; end

 book = Book.find( 1 )
 book.to_csv

Example 2, only exporting certain fields

1
2
3
4

 class Book < ActiveRecord::Base ; end

 book = Book.find( 1 )
 book.to_csv( :only=>%W( title isbn )

Example 3, exporting a model including a belongs_to association

1
2
3
4
5
6

 class Book < ActiveRecord::Base
   belongs_to :author
 end

 book = Book.find( 1 )
 book.to_csv( :include=>:author )

This also works for a has_one relationship. The :include option can also be an array of has_one/belongs_to associations. This by default includes all fields on the belongs_to association.

Example 4, exporting a model including a has_many association

1
2
3
4
5
6

 class Book < ActiveRecord::Base
   has_many :tags
 end

 book = Book.find( 1 )
 book.to_csv( :include=>:tags )

This by default includes all fields on the has_many assocaition. This can also be an array of multiple has_many relationships. The array can be mixed with has_one/belongs_to associations array as well. IE: :include=>[ :author, :sales ]

Example 5, nesting associations

1
2
3
4
5
6
7
8
9

 class Book < ActiveRecord::Base
   belongs_to :author
   has_many :tags
 end

 book = Book.find( 1 )
 book.to_csv( :include=>{
                 :author => { :only=>%W( name ) },
                 :tags => { :only=>%W( tagname ) } )

Example 6, using Arrays returned by find

1
2
3
4
5
6
7
8
9

 class Book < ActiveRecord::Base
   belongs_to :author
   has_many :tags
 end

 books = Book.find( :all )
 books.to_csv( :include=>{
                 :author => { :only=>%W( name ) },
                 :tags => { :only=>%W( tagname ) } )

Temporary Table Support

Example 1, using defaults

1
2
3

 class Project < ActiveRecord::Base ; end

 Project.create_temporary_table

This creates a temporary table named ‘temp_projects’ and creates a constant name TempProject. The table structure is copied from the projects table.

Example 2, using :table_name and :model options

  Project.create_temporary_table :table_name=>'my_projects', :model=>'MyProject'

This creates a temporary table named ‘my_projects’ and creates a constant named MyProject. The table structure is copied from the projects table.

Example 3, using :like

  ActiveRecord::Base.create_temporary_table :like=>Project

This is the same as calling Project.create_temporary_table.

Example 4, using block form

1
2
3

  Project.create_temporary_table do |t|
    # ...
  end

Using the block form will automatically drop the temporary table when the block exits. t which is passed into the block is the temporary table class. In the above example t equals TempProject. The block form can be used with all of the available options.

Foreign Key Support

Enables support for enabling and disabling foreign keys for the underlying database connection for ActiveRecord. This can be used with or without block form. This also uses the connection attached to the model.

Example 1, without block form

1
2

  Project.foreign_keys.disable
  Project.foreign_keys.enable

If you use this form you have to manually re-enable the foreign keys.

Example 2, with block form

1
2
3
4
5
6
7

  Project.foreign_keys.disable do
    # ...
  end

 Project.foreign_keys.enable do
   # ...
 end

If you use the block form the foreign keys are automatically enabled or disabled when the block exits. This currently does not restore the state of foreign keys to the state before the block was entered.

Download It Now!

ActiveRecord::Extensions 0.4.0

Check it out from SVN into your Rails project

svn co svn://rubyforge.org/var/svn/arext/tags/ar-extensions-0.4.0 vendor/plugins/