Recursively adding .org files in a top-level directory for org-agenda-files takes a long time

Question

I'm trying to find a way to quickly recurse through every subdirectory searching for org files. I've found several solutions (Elisp Cookbook, and several solutions on github), but they don't handle my real world usage (hundreds of directories (and subdirectories) and hundreds of org files). They seem to run forever on my system (Windows 7, with max-lisp-eval-depth = 10000). My work around is to add each directory manually to my org-agenda-list, but it's annoying and I know I've probably forgotten some. Any suggestions would be appreciated.

Answer 1

Haven't seen other people post this, so I will do. Have you tried load "find-list" library and use its "find-lisp-find-files" function? I added these lines in my org config and it works, but it may not fit your performance requirement:

(load-library "find-lisp") (setq org-agenda-files (find-lisp-find-files "FOLDERNAME" "\.org$"))

source: http://emacs-orgmode.gnu.narkive.com/n5bQRs5t/o-multiple-recursive-directories-with-org-agenda-files

Answer 2

The following code works well in emacs 24.3+:

;; Collect all .org from my Org directory and subdirs
(setq org-agenda-file-regexp "\\`[^.].*\\.org\\'") ; default value
(defun load-org-agenda-files-recursively (dir) "Find all directories in DIR."
    (unless (file-directory-p dir) (error "Not a directory `%s'" dir))
    (unless (equal (directory-files dir nil org-agenda-file-regexp t) nil)
      (add-to-list 'org-agenda-files dir)
    )
    (dolist (file (directory-files dir nil nil t))
        (unless (member file '("." ".."))
            (let ((file (concat dir file "/")))
                (when (file-directory-p file)
                    (load-org-agenda-files-recursively file)
                )
            )
        )
    )
)
(load-org-agenda-files-recursively "/path/to/your/org/dir/" ) ; trailing slash required

It does not require intermediate files creation and you can put it on a shortcut as well.

To be able to refile to any file found add this:

(setq org-refile-targets
      '((nil :maxlevel . 3)
        (org-agenda-files :maxlevel . 1)))

Answer 3

As of Emacs 25, you can use directory-files-recursively which returns all files matching a regex from a root directory.

(defun org-get-agenda-files-recursively (dir)
  "Get org agenda files from root DIR."
  (directory-files-recursively dir "\.org$"))

(defun org-set-agenda-files-recursively (dir)
  "Set org-agenda files from root DIR."
  (setq org-agenda-files 
    (org-get-agenda-files-recursively dir)))

(defun org-add-agenda-files-recursively (dir)
  "Add org-agenda files from root DIR."
  (nconc org-agenda-files 
    (org-get-agenda-files-recursively dir)))

(setq org-agenda-files nil) ; zero out for testing

(org-set-agenda-files-recursively "~/Github") ; test set

(org-add-agenda-files-recursively "~/Dropbox") ; test add 

You can view the contents of org-agenda-files by typing C-hv org-agenda-files.

Answer 4

Little late to the party, and probably not the answer you were hoping for, but the only way I found to speed up my agenda load times was by not including some directories which had thousands of org files.

(setq org-directory "~/org/"
      my-agenda-dirs '("personal" "projects" "todos" "work")
      org-agenda-files (mapcan (lambda (x) (directory-files-recursively
                                             (expand-file-name x org-directory)
                                             "\.org$"))
                               my-agenda-dirs))

Essentially rather than recursing over my entire org dir I only recurse through a select few of it's subdirs.

Answer 5

I also tried "recursive directory listing" at Emacs startup. It simply is way to loooong to be usable. So, try to stick with a limited number of "root" directories where you put your agenda files.

Even better is sticking with "~/org", and possibly a few subdirs like "work" and "personal", and that's it. You can put there your agenda files, and have links Inside them to your real project root dirs.

I know, this is not optimal, but I don't have anything better right now.