Skip to Content

Drupal

.VDMi/Blog: I went to Drupal 7.33 and all I got was a slow site

Planet Drupal - 27 November 2014 - 1:29pm
So, you just upgraded your site(s) to Drupal >= 7.33, everything seemed fine in your tests. You deployed the new release and after a while you notice that your site isn't as fast as before. It's actually slowing down everything on your server. You started Googling and ended up on this blogpost. Sounds like your story? Read on!

I spent the last 2 hours figuring this out, I decided it would be best to write it up right away while it's fresh in my memory. TLDR; at the bottom of this post.

We did an upgrade to Drupal 7.34 recently, we thought everything was fine. Release went over 3 different environments before deploying to live and no actual issues were found.

After deployment to live, we got some reports that the site was responding slow. We didn't directly link it to the upgrade to 7.34, I actually did a diff between 7.32 and 7.34 to see what changed after these reports and did not see anything suspicious that could cause this.

We had to investigate after a while as there was no sign of improvement, the CPU of the server was hitting 100% 24/7. New Relic monitoring told us about many calls to the "file_scan_directory" function of Drupal. When I actually logged the calls with the following snippet:

static $count;
if (!isset($count)) {
  $count = 0;
}
$count++;
drupal_debug($count . PHP_EOL);

The count actually went up to 700 for every request (It's quite a large project, plus the file_scan_directory is recursive).
When I printed the "debug_backtrace", I saw that this call was coming from "drupal_get_filename".
Looking at the function arguments, Drupal was looking for a imagecache_actions file, why?! And why on every request? Doesn't it cache these records in the registry?!

Yes it does! It appeared the imagecache_actions module had a typo in it (patch here):

module_load_include('inc', 'imagcache_actions', 'image_overlay.inc');

This should be:

module_load_include('inc', 'imagecache_actions', 'image_overlay.inc');

This would not have been an issue previously, 7.33 introduced a change though.
Pre 7.33:

$file = db_query("SELECT filename FROM {system} WHERE name = :name AND type = :type", array(':name' => $name, ':type' => $type))->fetchField();
if (file_exists(DRUPAL_ROOT . '/' . $file)) {
  $files[$type][$name] = $file;
}

7.33 or higher:

$file = db_query("SELECT filename FROM {system} WHERE name = :name AND type = :type", array(':name' => $name, ':type' => $type))->fetchField();
if ($file !== FALSE && file_exists(DRUPAL_ROOT . '/' . $file)) {
  $files[$type][$name] = $file;
}

Before 7.33, Drupal would try to find the record in the system table, it wouldn't find it and the $file would be NULL. The resulting string would be "DRUPAL_ROOT . '/' . $file", as $file is NULL, you can also see it as "DRUPAL_ROOT . '/'". Obviously the DRUPAL_ROOT exists, so it returns TRUE. It would put the record in the $files array and continue with what it was doing.

Because 7.33 and higher does a NULL-check on $file, it will not add any record to the $files array. This causes it to go into the file discovery routine:

if (!isset($files[$type][$name])) {
  // ... Some code
  $matches = drupal_system_listing("/^" . DRUPAL_PHP_FUNCTION_PATTERN . "\.$extension$/", $dir, 'name', 0);
  foreach ($matches as $matched_name => $file) {
    $files[$type][$matched_name] = $file->uri;
  }
}

This code will try to scan your Drupal installation for the given file. It will not find the file and continue eventually, but it will execute the file search in EVERY request that you execute the module_load_include.

While our issue was in the imagecache_actions module, your issue might be in any module (even custom) which does a wrong module_load_include.
It's very hard to find this out yourself. You can edit includes/bootstrap.inc on line 866 to write some info away to /tmp/drupal_debug.txt:
Add the following code after line 866:

else {
  drupal_debug('Missing file ' . $type . ' ' . $name . ' ' . DRUPAL_ROOT . '/' . $file . PHP_EOL);
}

TLDR; an issue in imagecache_actions combined with an upgrade to Drupal >= 7.33 killed the performance of our site. Patch for imagecache_actions here. Causing issue/patch here.

Categories: Drupal

Blair Wadman: Eleven tips to keep Drupal up to date with security releases

Planet Drupal - 27 November 2014 - 12:36pm

Keeping your Drupal site up to date has always been of critical importance to ensure it remains secure. Last month's announcement of a SQL Injection vulnerability and subsequent announcement of automated attacks within 7 hours caused wide spread panic across much of the Drupal community.

Tags: DrupalPlanet Drupal
Categories: Drupal

Appnovation Technologies: Searching and attaching images to content

Planet Drupal - 27 November 2014 - 9:00am

Because of its ability to extend the core platform, Drupal has become a popular CMS/Framework for many large media and publishing companies.

var switchTo5x = false;stLight.options({"publisher":"dr-75626d0b-d9b4-2fdb-6d29-1a20f61d683"});
Categories: Drupal

Oliver Davies: Include environment-specific settings files on Pantheon

Planet Drupal - 27 November 2014 - 8:24am

I was recently doing some work on a site hosted on Pantheon and came across an issue, for which part of the suggested fix was to ensure that the $base_url variable was explicitly defined within settings.php (this is also best practice on all Drupal sites).

The way that was recommended was by using a switch() function based on Pantheon's environment variable. For example:

Tags:
Categories: Drupal

Cheppers blog: Acquia Certified Developer exam - passed!

Planet Drupal - 27 November 2014 - 4:21am

We are proud to announce that Cheppers has now three Acquia Certified Developers!
This Monday Mau, Attila and Andor have all passed the exam held by Acquia, and we are very proud of them.

Categories: Drupal

Kristian Polso: Crawling the top 15,000 Drupal websites

Planet Drupal - 27 November 2014 - 2:01am
So I crawled the top 1,000,000 websites from Alexa, looking for all of the Drupal websites (and other popular CMS's). Here are the results.
Categories: Drupal

Joachim's blog: A git-based patch workflow for drupal.org (with interdiffs for free!)

Planet Drupal - 27 November 2014 - 12:39am

There's been a lot of discussion about how we need github-like features on d.org. Will we get them? There's definitely many improvements in the pipeline to the way our issue queues work. Whether we actually need to replicate github is another debate (and my take on it is that I don't think we do).

In the meantime, I think that it's possible to have a good collaborative workflow with what we have right now on drupal.org, with just the issue queue and patches, and git local branches. Here's what I've gradually refined over the years. It's fast, it helps you keep track of things, and it makes the most of git's strengths.

A word on local branches

Git's killer feature, in my opinion, is local branches. Local branches allow you to keep work on different issues separate, and they allow you to experiment and backtrack. To get the most out of git, you should be making small, frequent commits.

Whenever I do a presentation on git, I ask for a show of hands of who's ever had to bounce on CMD-Z in their text editor because they broke something that was working five minutes ago. Commit often, and never have that problem again: my rule of thumb is to commit any time that your work has reached a state where if subsequent changes broke it, you'd be dismayed to lose it.

Starting work on an issue

My first step when I'm working on an issue is obviously:

  git pull

This gets the current branch (e.g. 7.x, 7.x-2.x) up to date. Then it's a good idea to reload your site and check it's all okay. If you've not worked on core or the contrib project in question in a while, then you might need to run update.php, in case new commits have added updates.

Now start a new local branch for the issue:

  git checkout -b 123456-foobar-is-broken

I like to prefix my branch name with the issue number, so I can always find the issue for a branch, and find my work in progress for an issue. A description after that is nice, and as git has bash autocompletion for branch names, it doesn't get in the way. Using the issue number also means that it's easy to see later on which branches I can delete to unclutter my local git checkout: if the issue has been fixed, the branch can be deleted!

So now I can go ahead and start making commits. Because a local branch is private to me, I can feel free to commit code that's a total mess. So something like:

  dpm($some_variable_I_needed_to_examine);
  /*
  // Commented-out earlier approach that didn't quite work right.
  $foo += $bar;
  */
  // Badly-formatted code that will need to be cleaned up.
  if($badly-formatted_code) { $arg++; }

That last bit illustrates an important point: commit code before cleaning up. I've lost count of the number of times that I've got it working, and cleaned up, and then broken it because I've accidentally removed an important that was lost among the cruft. So as soon as code is working, I make a commit, usually whose message is something like 'TOUCH NOTHING IT WORKS!'. Then, start cleaning up: remove the commented-out bits, the false starts, the stray code that doesn't do anything. (This is where you find it actually does, and breaks everything: but that doesn't matter, because you can just revert to a previous commit, or even use git bisect.)

Keeping up to date

Core (or the module you're working on) doesn't stay still. By the time you're ready to make a patch, it's likely that there'll be new commits on the main development branch (with core it's almost certain). And prior to that, there may be commits that affect your work in some way: API changes, bug fixes that you no longer need to work around, and so on.

Once you've made sure there's no work currently uncommitted (either use git stash, or just commit it!), do:

git fetch
git rebase BRANCH

where BRANCH is the main development branch that is being committed to on drupal.org, such as 8.0.x, 7.x-2.x-dev, and so on.

(This is arguably one case where a local branch is easier to work with than a github-style forked repository.)

There's lots to read about rebasing elsewhere on the web, and some will say that rebasing is a terrible thing. It's not, when used correctly. It can cause merge conflicts, it's true. But here's another place where small, regular commits help you: small commits mean small conflicts, that shouldn't be too hard to resolve.

Making a patch

At some point, I'll have code I'm happy with (and I'll have made a bunch of commits whose log messages are 'clean-up' and 'formatting'), and I want to make a patch to post to the issue:

  git diff 7.x-1.x > 123456.PROJECT.foobar-is-broken.patch

Again, I use the issue number in the name of the patch. Tastes differ on this. I like the issue number to come first. This means it's easy to use autocomplete, and all patches are grouped together in my file manager and the sidebar of my text editor.

Reviewing and improving on a patch

Now support Alice comes along, reviews my patch, and wants to improve it. She should make her own local branch:

  git checkout -b 123456-foobar-is-broken

and download and apply my patch:

  wget PATCHURL
  patch -p1 < 123456.PROJECT.foobar-is-broken.patch

(Though I would hope she has a bash alias for 'patch -p1' like I do. The other thing to say about the above is that while wget is working at downloading the patch, there's usually enough time to double-click the name of the patch in its progress output and copy it to the clipboard so you don't have to type it at all.)

And finally commit it to her branch. I would suggest she uses a commit message that describes it thus:

  git commit -m "joachim's patch at comment #1"

(Though again, I would hope she uses a GUI for git, as it makes this sort of thing much easier.)

Alice can now make further commits in her local branch, and when she's happy with her work, make a patch the same way I did. She can also make an interdiff very easily, by doing a git diff against the commit that represents my patch.

Incorporating other people's changes to ongoing work

All simple so far. But now suppose I want to fix something else (patches can often bounce around like this, as it's great to have someone else to spot your mistakes and to take turns with). My branch looks like it did at my patch. Alice's patch is against the main branch (for the purposes of this example, 7.x-1.x).

What I want is a new commit on the tip of my local branch that says 'Alice's changes from comment #2'. What I need is for git to believe it's on my local branch, but for the project files to look like the 7.x-1.x branch. With git, there's nearly always a way:

  git checkout 7.x-1.x .

Note the dot at the end. This is the filename parameter to the checkout command, which tells git that rather than switch branches, you want to checkout just the given file(s) while staying on your current branch. And that the filename is a dot means we're doing that for the entire project. The branch remains unchanged, but all the files from 7.x-1.x are checked out.

I can now apply Alice's patch:

  wget PATCHURL
  patch -p1 < 123456.2.PROJECT.foobar-is-broken.patch

(Alice has put the comment ID after the issue ID in the patch filename.)

When I make a commit, the new commit goes on the tip of my local branch. The commit diff won't look like Alice's patch: it'll look like the difference between my patch and Alice's patch: effectively, an interdiff.

  git commit -m "Alice's patch at comment #2"

I can now do a diff as before, post a patch, and work on the issue advances to another iteration.

Here's an example of my local branch for an issue on Migrate I've been working on recently. You can see where I made a bunch of commits to clean up the documentation to get ready to make a patch. Following that is a commit for the patch the module maintainer posted in response to mine. And following that are a few further tweaks that I made on top of the maintainer's patch, which I then of course posted as another patch.

Improving on our tools

Where next? I'm pretty happy with this workflow as it stands, though I think there's plenty of scope for making it easier with some git or bash aliases. In particular, applying Alice's patch is a little tricky. (Though the stumbling block there is that you need to know the name of the main development branch. Maybe pass the script the comment URL, and let it ask d.org what the branch of that issue is?)

Beyond that, I wonder if any changes can be made to the way git works on d.org. A sandbox per issue would replace the passing around of patch files: you'd still have your local branch, and merge in and push instead of posting a patch. But would we have one single branch for the issue's development, which then runs the risk of commit clashes, or start a new branch each time someone wants to share something, which adds complexity to merging? And finally, sandboxes with public branches mean that rebasing against the main project's development can't be done (or at least, not without everyone know how to handle the consequences). The alternative would be merging in, which isn't perfect either.

The key thing, for me, is to preserve (and improve) the way that so often on d.org, issues are not worked on by just one person. They're a ball that we take turns pushing forward (snowball, Sisyphean rock, take your pick depending on the issue!). That's our real strength as a community, and whatever changes we make to our toolset have to be made with the goal of supporting that.

Categories: Drupal

PreviousNext: Lightning talk - Drupal 8's Third Party Settings Interface

Planet Drupal - 26 November 2014 - 6:22pm

During this weeks developers' meeting our lightning talk was all about Drupal 8's ThirdPartySettingsInterface.

Here's the video introduction to this powerful new feature in Drupal.

Categories: Drupal

Acquia: Part 2 – Cal Evans and Jeffrey A. "jam" McGuire talk open source

Planet Drupal - 26 November 2014 - 3:53pm
Language Undefined

Voices of the ElePHPant / Acquia Podcast Ultimate Cage Match Part 2 - I had the chance to try to pull Cal Evans out of his shell at DrupalCon Amsterdam. After a few hours, he managed to open up and we talked about a range of topics we have in common. In this part of our conversation we talk about 'Getting off the Island', inter-project cooperation in PHP and Drupal's role in that; the reinvention and professionalization of PHP core development; decoupled, headless Drupal 8; PHP and the LAMP stack as tools of empowerment and the technologists' responsibility to make devices and applications that are safe, secure, and private by default.

Categories: Drupal

Alternative Language

New Drupal Modules - 26 November 2014 - 12:48pm

Allows a language to specify an alternative language so if a node translation for a specific language does not exist, Drupal will try to load the translation for the alternative language before throwing a 404.

This module is useful if you have multiple languages and only one language is typically translated. Rather than create all of the other translations and maintain those, the non-translated languages can load the node from the alternative language.

Categories: Drupal

Set Front Page

New Drupal Modules - 26 November 2014 - 11:37am

A simple module to make the current page the front page.

Categories: Drupal

Four Kitchens: Extracting data from Drupal entities the right way

Planet Drupal - 26 November 2014 - 10:54am

If you’ve ever had to extract data from Drupal entities you know it can be a painful process. This post presents a solution for distilling Drupal entities into human-readable documents that can be easily consumed by other modules and services.

Projects Drupal
Categories: Drupal

Access Control Bridge

New Drupal Modules - 26 November 2014 - 10:48am
Description

The Drupal community has developed several excellent modules to get fine-grained access control over a node (e.g. Content Access, Domain Access, Workflow, Organic Groups, Taxonomy Access Control, ...)

Categories: Drupal

Code Karate: Drupal 7 File Resumable Upload Module

Planet Drupal - 26 November 2014 - 7:04am
Episode Number: 181

The Drupal 7 File Resumable Upload Module is a great way to allow your Drupal site to upload large files. This is especially helpful if your server limits the size of files you can upload to your Drupal site. The module simply replaces the standard Drupal file upload field with a better alternative that allows:

Tags: DrupalDrupal 7File ManagementMediaDrupal Planet
Categories: Drupal

Views Toggle Filter

New Drupal Modules - 26 November 2014 - 4:13am

This exposed filter does not filter anything. It just shows and hides other exposed filters when you click either of two radio buttons which can be labeled as "mini search" and "extended search".

Categories: Drupal

Paul Booker: Programmatically adding terms into a vocabulary from a structured text file

Planet Drupal - 26 November 2014 - 2:06am
/** * Implements hook_install(). */ function artist_install() { artist_install_vocabularies(); artist_install_terms(); } /** * Installs artist module's default terms that are read from * text files in the module's includes folder. */ function artist_install_terms() { foreach (array_keys(artist_vocabularies()) as $machine_name) { $v = taxonomy_vocabulary_machine_name_load($machine_name); $wrapper = entity_metadata_wrapper('taxonomy_vocabulary', $v); if ($wrapper->term_count->value() == 0) { $path = drupal_get_path('module', 'artist') . '/includes/terms_' . $v->machine_name . '.txt'; $lines = file($path, FILE_SKIP_EMPTY_LINES); artist_install_term_tree($wrapper, $lines); } } } /** * Installs a term tree. * @param $vwrapper * EntityMetadataWrapper of a taxonomy_vocabulary entity. * @param $lines * Array of lines from the term text file. The iterator must be set * to the line to parse. * @param $last * Either NULL or the parent term ID. * @param $depth * Current depth of the tree. */ function artist_install_term_tree($vwrapper, &$lines, $last = NULL, $depth = 0) { $wrapper = NULL; while ($line = current($lines)) { $name = trim($line); $line_depth = max(strlen($line) - strlen($name) - 1, 0); if ($line_depth < $depth) { return; } else if ($line_depth > $depth) { $tid = $wrapper ? $wrapper->tid->value() : NULL; artist_install_term_tree($vwrapper, $lines, $tid, $depth+1); } else { $data = array( 'name' => $name, 'vid' => $vwrapper->vid->value(), 'parent' => array($last ? $last : 0), ); $term = entity_create('taxonomy_term', $data); $wrapper = entity_metadata_wrapper('taxonomy_term', $term); $wrapper->save(); next($lines); } } } /** * Installs terms into default vocabularies. */ function artist_update_7201(&$sandbox) { artist_install_terms(); }

In the preceding code, term names are read from text files that have tab indentation to symbolize the term hierarchy

Tags:
Categories: Drupal

Masonry Panels

New Drupal Modules - 25 November 2014 - 9:32pm

Provides 'Masonry' style layouts as an option for rendering the contents of panel panes.
Masonry rendering will re-pack panes in your panel to minimize vertical space and provide column-like appearance while the items are still in natural (horizontal) order on the page.

Categories: Drupal

Drupal Association News: Drupal Association Board Meeting: 21 November 2014

Planet Drupal - 25 November 2014 - 8:27pm

It is hard to believe, but we just finished our second-to-last board meeting of the year. The Association has grown and changed so much in 2014 and the November meeting was a great chance to talk about some of those changes and what we are planning for 2015. It was a somewhat short public meeting as we spent the bulk of our time in Executive Session to review the financial statements from the last quarter and the staff's proposed 2015 Leadership Plan and Budget. As always, you can review the minutes, the materials, or the meeting recording to catch up on all the details, and here's a summary for you as well.

Staff Update

DrupalCons: DrupalCon Amstedam is over, and we are now focused on evaluation of the event - reviewing all the session and conference evaluations as well as closing up the financials for the event. We will have an in-depth review of the event at the December board meeting. Next up is DrupalCon Latin America, which is progressing nicely now with sessions accepted and registration open. One final DrupalCon note is that DrupalCon Los Angeles session submissions should open in January, so mark your calendars for that.

Drupal.org: Drupal.org has been our primary imperative at the Association this year. We've spent 2014 building a team and really working to pay off a mountain of accumulated technical debt while also balancing the need to release new features for the community. We are very pleased that, with the working groups and community feedback, we've been able to release a Drupal.org roadmap for 2015. We also released a new Terms of Service and Privacy Policy after extensive edits based on community feedback. We'll continue to respond to questions and ideas about these documents and make notes for future releases. We have also finally been able to deploy and use a suite of over 400 tests on Drupal.org. This is work that was initiated by Melissa Anderson (eliza411) and she was extremely helpful in getting those tests up and running again. We're thrilled to be using this contribution after all this time and are extremely grateful to Melissa.

Community Programs: We just held our final Global Training Days for 2014 with over 80 companies on every continent but Antarctica (c'mon penguins!). This program has continued to evolve with new partnerships and currciulums used this time around, as well as a plethora of great photos and tweets sent our way.

Marketing and Communications: Joe Saylor and our team have been working with the Security team to develop a follow up to the recent security announcement focused on the lessons learned and changes our community has made in response to the situation. It's another great example of an Association and community volunteer partnership.

Licensing Working Group

As we discussed in the August board meeting, some community members have expressed concern over the slow and sometimes inconsistent response to licensing issues on Drupal.org. In response, a volunteer group came together to draft a charter which was published for comment just after DrupalCon Amsterdam. Some of the main points from the charter include:

  • All members (chair + 4-5) appointed by the Board

  • Scope is limited to licensing of code and other assets on D.O only, not other sites, and not determining WHICH license is used

  • Group responds to issues, does not police for issues

  • Will maintain the whitelist of allowable assets

The Association board voted to adopt the charter, so our next steps are to recruit members, create a queue for licesning issues, and then provide some legal training for our new Working Group members. If you are interested in participating, you can nominate yourself for the Working Group. We are planning to present a slate of candidates to the board for approval in the January 2015 meeting.

3rd Quarter Financials

Once per quarter, the board discusses the previous quarter's financial statements and then votes to approve and publish them. In the November meeting the board approved the Q3 financials:

I recently wrote a post highlighting how to read our financial statments, but will summarize here for you as well. Generally speaking, we are performing well ahead of our budgeted deficit spend. Though we had planned for a -$750,000 budget for 2014, a combination of slow tech team hiring, savings on Drupal.org contractor expenses, and some better than budgeted revenue means that we will not operate at nearly that level of loss for 2014. Instead the burden of the staffing investment we've made will really be felt in 2015. We'll see more of this and have a larger discussion when we release our budget and leadership plan next month.

As always, please let me know if you have any questions or share your thoughts in the comments.

Flickr phtoo: steffen.r

Categories: Drupal

Drupal core announcements: Drupal 8 beta 4 on Wednesday, December 17, 2014

Planet Drupal - 25 November 2014 - 5:54pm

The next beta for Drupal 8 will be beta 4! Here is the schedule for the beta release.

Tuesday, December 16, 2014 Only critical and major patches committed Wednesday, December 17, 2014 Drupal 8.0.0-beta4 released. Emergency commits only.
Categories: Drupal

Configuration Packager

New Drupal Modules - 25 November 2014 - 3:27pm

Configuration Packager for Drupal 8 enables the packaging of site configuration into modules, like Features for Drupal 7. Rather than producing manually authored individual features, Configuration Packager analyzes the site configuration and automatically divides it up into configuration modules based on configured preferences.

Categories: Drupal
Syndicate content


Google+
about seo