Backup / Mirror Github repositories_问答_开发者

I'd like to peri开发者_运维知识库odically create a backup of my github repositories. Is there a quick way to pull all of them without knowing what the entire list is?

Walter

You can get the entire list via GitHub's API:

curl http://github.com/api/v2/yaml/repos/show/walterjwhite

For example, this simple DOS/Unix shell one-liner:

ruby -ryaml -ropen-uri -e "puts YAML.load(open('http://github.com/api/v2/yaml/repos/show/walterjwhite'))['repositories'].map {|r| %Q[* **#{r[:name]}** (#{r[:description]}) is at <#{r[:url]}/>] }"

prints (assuming you have Ruby installed):

logback (The reliable, generic, fast and flexible logging framework for Java.) is at http://github.com/walterjwhite/logback/
Test-Automation (provides UI Test Automation framework) is at http://github.com/walterjwhite/Test-Automation/
model.base (Base Model Project) is at http://github.com/walterjwhite/model.base/
project.configuration (Project Configuration (Jalopy, Checkstyle, etc.)) is at http://github.com/walterjwhite/project.configuration/
model.http (Http Models / Enties) is at http://github.com/walterjwhite/model.http/
model.address (Address models and entities) is at http://github.com/walterjwhite/model.address/
model.geolocation.base (Geolocation models and entities) is at http://github.com/walterjwhite/model.geolocation.base/
model.user (User models and entities) is at http://github.com/walterjwhite/model.user/
model.annotations.current (Used to inject the current date / user into an entity) is at http://github.com/walterjwhite/model.annotations.current/
model.annotations.deidentify (Used to deidentify PHI fields) is at http://github.com/walterjwhite/model.annotations.deidentify/
model.annotations.verify (Used to verify fields (email address, address, phone number, etc.)) is at http://github.com/walterjwhite/model.annotations.verify/
model.time (Time entities) is at http://github.com/walterjwhite/model.time/
model.http-security (Http Security entities) is at http://github.com/walterjwhite/model.http-security/
model.restriction.base (Base Restriction Models) is at http://github.com/walterjwhite/model.restriction.base/
model.restriction.time (Time Restrictions) is at http://github.com/walterjwhite/model.restriction.time/
model.restriction.user (User/Group/Role Restrictions) is at http://github.com/walterjwhite/model.restriction.user/
model.web.base (Base Web Model) is at http://github.com/walterjwhite/model.web.base/
model.web.resource (Web Resource Model Project) is at http://github.com/walterjwhite/model.web.resource/
model.web.file (File model) is at http://github.com/walterjwhite/model.web.file/
model.web.content (Web Content Models) is at http://github.com/walterjwhite/model.web.content/
model.deployment (Deployment models / entities) is at http://github.com/walterjwhite/model.deployment/
model.geolocation.tracking (Geolocation Tracking) is at http://github.com/walterjwhite/model.geolocation.tracking/
model.geolocation.device-location (Device Location models) is at http://github.com/walterjwhite/model.geolocation.device-location/
model.defect (Defect models / entities) is at http://github.com/walterjwhite/model.defect/
model.test.error (Test Errors) is at http://github.com/walterjwhite/model.test.error/
model.web.automation.history (Test History) is at http://github.com/walterjwhite/model.web.automation.history/
model.html (Test Html) is at http://github.com/walterjwhite/model.html/
model.test.user (Test Http) is at http://github.com/walterjwhite/model.test.user/
model.web.automation.test (Test Test entities) is at http://github.com/walterjwhite/model.web.automation.test/
model.web.automation.validation (Test Validation entities) is at http://github.com/walterjwhite/model.web.automation.validation/
service.query.base (Provides framework for automatically injecting queries from an interface definition + annotation) is at http://github.com/walterjwhite/service.query.base/
service.query.defect (Provides query service for querying against defects / fixes) is at http://github.com/walterjwhite/service.query.defect/
service.query.deployment (Deployment query service) is at http://github.com/walterjwhite/service.query.deployment/
model.restriction.geolocation (Geolocation restrictions) is at http://github.com/walterjwhite/model.restriction.geolocation/
service.query.address (Address query servicd) is at http://github.com/walterjwhite/service.query.address/
service.query.geolocation.device-location (Device Location Query Service) is at http://github.com/walterjwhite/service.query.geolocation.device-location/
service.query.geolocation.base (Geolocation Query Service) is at http://github.com/walterjwhite/service.query.geolocation.base/
service.query.http (Http Query Service) is at http://github.com/walterjwhite/service.query.http/
service.query.http-security (Http Security query service) is at http://github.com/walterjwhite/service.query.http-security/
service.query.user (User Query Service) is at http://github.com/walterjwhite/service.query.user/
service.query.web.file (Web File Query Service) is at http://github.com/walterjwhite/service.query.web.file/
service.query.web.content (Web Content Query Service) is at http://github.com/walterjwhite/service.query.web.content/
observer.startup.deployment (Startup Event Observer (deployment)) is at http://github.com/walterjwhite/observer.startup.deployment/
observer.startup.defect (Startup Event Observer (defect)) is at http://github.com/walterjwhite/observer.startup.defect/
listener.servlet.request (Request Servlet Listener) is at http://github.com/walterjwhite/listener.servlet.request/
util.network (Network utilities) is at http://github.com/walterjwhite/util.network/
listener.servlet.geolocation (Geolocation Servlet Listener) is at http://github.com/walterjwhite/listener.servlet.geolocation/
listener.persistence.audit (Audit Persistence Listeners (Hibernate)) is at http://github.com/walterjwhite/listener.persistence.audit/
listener.persistence.current (Current (User / Date) Persistence Listeners) is at http://github.com/walterjwhite/listener.persistence.current/
listener.persistence.restriction (Restriction Persistence Listener (secures entities on load)) is at http://github.com/walterjwhite/listener.persistence.restriction/
service.geolocation (Geolocation update service) is at http://github.com/walterjwhite/service.geolocation/
service.notification (Notification Service) is at http://github.com/walterjwhite/service.notification/
service.traffic-analysis (Traffic Analysis Service) is at http://github.com/walterjwhite/service.traffic-analysis/
servlet.file (File Servlet) is at http://github.com/walterjwhite/servlet.file/
util.web.file (File Utilities) is at http://github.com/walterjwhite/util.web.file/
servlet.filter.base (Base Servlet Filters) is at http://github.com/walterjwhite/servlet.filter.base/
servlet.filter.web.content (Web Content servlet filter) is at http://github.com/walterjwhite/servlet.filter.web.content/
servlet.filter.security (Security servlet filters) is at http://github.com/walterjwhite/servlet.filter.security/
io.file (File functions (deprecated)) is at http://github.com/walterjwhite/io.file/
deprecated.listener.postInitialization.libraryLogger (Library Logger (deprecated)) is at http://github.com/walterjwhite/deprecated.listener.postInitialization.libraryLogger/
deprecated.listener.startup.databaseImporter (Database Importer (allows multiple import.sql scripts to be run on server start)) is at http://github.com/walterjwhite/deprecated.listener.startup.databaseImporter/
deprecated.listener.startup.scheduler (Scheduler (enables jobs to be schedule simply with annotations)) is at http://github.com/walterjwhite/deprecated.listener.startup.scheduler/
docbook (Docbook format (provides docbook configuration for standard formatting)) is at http://github.com/walterjwhite/docbook/
deprecated.model.client (Client models) is at http://github.com/walterjwhite/deprecated.model.client/
deprecated.model.virtualhost (Virtualhost models) is at http://github.com/walterjwhite/deprecated.model.virtualhost/
util.web.resource (Web Resource utils (deprecated, use security listeners instead for transparent security of entities)) is at http://github.com/walterjwhite/util.web.resource/
servlet.filter.notification (Notification servlet filter) is at http://github.com/walterjwhite/servlet.filter.notification/
spring-core (base project for spring projects (deprecated)) is at http://github.com/walterjwhite/spring-core/
view.content (JSF components for content projects (provides the view layer)) is at http://github.com/walterjwhite/view.content/
documentation.server-setup (Documentation for setting up a server (trying to make a repeatable process and ensure fewer defects and potential vulnerabilities)) is at http://github.com/walterjwhite/documentation.server-setup/
server.http (Http Server using JBoss Netty) is at http://github.com/walterjwhite/server.http/
listener.persistence.deidentify (Deidentification persistence listener) is at http://github.com/walterjwhite/listener.persistence.deidentify/
util.http (Http Utilities) is at http://github.com/walterjwhite/util.http/
deprecated.service.translate (Translation Service) is at http://github.com/walterjwhite/deprecated.service.translate/
html5.example.danforys.mesmerizer (HTML5 grid example) is at http://github.com/walterjwhite/html5.example.danforys.mesmerizer/
html5.framework (HTML5 framework (caching, client renderering, etc.)) is at http://github.com/walterjwhite/html5.framework/
documentation.os.open-solaris (Open Solaris Documentation) is at http://github.com/walterjwhite/documentation.os.open-solaris/
documentation.development (Development Documentation) is at http://github.com/walterjwhite/documentation.development/
interceptor.exception (Generic exception interceptor for CDI) is at http://github.com/walterjwhite/interceptor.exception/
observer.http-security (Http Security Observer (authentication events)) is at http://github.com/walterjwhite/observer.http-security/
model.preference.base (Base Preference model project) is at http://github.com/walterjwhite/model.preference.base/
model.preference.notification (Notification Preference Models) is at http://github.com/walterjwhite/model.preference.notification/
deprecated.web.base (Deprecated project (abstract web components)) is at http://github.com/walterjwhite/deprecated.web.base/
model.annotations.ui (Annotations for automatically rendering Web Action Entities) is at http://github.com/walterjwhite/model.annotations.ui/
model.web.action.geolocation (Geolocation web action models) is at http://github.com/walterjwhite/model.web.action.geolocation/
listener.servlet.web.traffic (Web Traffic Servlet Listener (captures web traffic)) is at http://github.com/walterjwhite/listener.servlet.web.traffic/
model.web.traffic (Web Traffic models (for capturing HTML5 traffic, can be used for HTML4 as well)) is at http://github.com/walterjwhite/model.web.traffic/
iterator.sql (SQL Iterator (for iterating through SQL scripts)) is at http://github.com/walterjwhite/iterator.sql/
model.notification (Notification Models/Entities) is at http://github.com/walterjwhite/model.notification/
model.fmea (FMEA models (for capturing potential failure mode and effects analysis)) is at http://github.com/walterjwhite/model.fmea/
service.web.automation.operator (Operator for web automation project (simulates an application user)) is at http://github.com/walterjwhite/service.web.automation.operator/
deprecated.iterator.web-automation (Web Automation iterators (page navigation)) is at http://github.com/walterjwhite/deprecated.iterator.web-automation/
deprecated.OS.gentoo.install (Gentoo Install scripts (deprecated)) is at http://github.com/walterjwhite/deprecated.OS.gentoo.install/
web.action.base (Base Web Action module) is at http://github.com/walterjwhite/web.action.base/
web.action.contact.email (Provides the web action for contacting the web site via email) is at http://github.com/walterjwhite/web.action.contact.email/
web.action.geolocation (Provides geolocation web action (type in an ip address, get back a geolocation)) is at http://github.com/walterjwhite/web.action.geolocation/
web.action.search (Search web action (ability to search through all entities provided the user has permission)) is at http://github.com/walterjwhite/web.action.search/
web.action.user (User web actions (for managing users)) is at http://github.com/walterjwhite/web.action.user/
web.service.geolocation (Geolocation web service) is at http://github.com/walterjwhite/web.service.geolocation/
service.query.web.resource (Web Resource query service) is at http://github.com/walterjwhite/service.query.web.resource/
model.web.action.contact.email (Email backing the Contact action) is at http://github.com/walterjwhite/model.web.action.contact.email/
interceptor.web (intercepts web action/service methods for auditing purposes) is at http://github.com/walterjwhite/interceptor.web/
prettyfaces.extension.sitemap (Prettyfaces sitemap extension (automatically generates a sitemap and publishes it to google, yahoo, msn, etc.)) is at http://github.com/walterjwhite/prettyfaces.extension.sitemap/
prettyfaces-tests (Some PrettyFaces test webapps) is at http://github.com/walterjwhite/prettyfaces-tests/
scheduler.quartz (Quartz Job Scheduler manager (initializes a quartz job scheduler ...)) is at http://github.com/walterjwhite/scheduler.quartz/
observer.rFactor (rFactor event observers (for managing an rFactor server)) is at http://github.com/walterjwhite/observer.rFactor/
observer.event-planner (Event Planner observers) is at http://github.com/walterjwhite/observer.event-planner/
web.action.news (News web action components) is at http://github.com/walterjwhite/web.action.news/
web.action.rFactor (rFactor web actions (managing rFactor servers, configurations, races, etc.)) is at http://github.com/walterjwhite/web.action.rFactor/
web.action.contact.chat (Chat web action components) is at http://github.com/walterjwhite/web.action.contact.chat/
service.automation.web.search.job (Job Search Web Automation Service) is at http://github.com/walterjwhite/service.automation.web.search.job/
deprecated.service.analytics (analytics service (deprecated, use an actual reporting package instead)) is at http://github.com/walterjwhite/deprecated.service.analytics/
service.backup (Backup Service (backs up a database, log files, etc.)) is at http://github.com/walterjwhite/service.backup/
model.web.action.bookmark (bookmark web action models) is at http://github.com/walterjwhite/model.web.action.bookmark/
model.web.action.search (search models) is at http://github.com/walterjwhite/model.web.action.search/
model.web.action.decision (Decisicion models (for capturing decisions and outcomes, useful for design decisions, and of course endless possibilities)) is at http://github.com/walterjwhite/model.web.action.decision/
model.weather (weather models) is at http://github.com/walterjwhite/model.weather/
model.recipe (recipe models) is at http://github.com/walterjwhite/model.recipe/
model.web.action.search.job (Job Search web action models) is at http://github.com/walterjwhite/model.web.action.search.job/
model.web.action.task (Task web action models) is at http://github.com/walterjwhite/model.web.action.task/
model.travel (Travel Models) is at http://github.com/walterjwhite/model.travel/
model.events (Events model) is at http://github.com/walterjwhite/model.events/
model.lodging (Lodging model) is at http://github.com/walterjwhite/model.lodging/
model.business.billing (Billing Business Models (for capturing billing information)) is at http://github.com/walterjwhite/model.business.billing/
model.business.business (Business Models (for capturing billing information)) is at http://github.com/walterjwhite/model.business.business/
model.business.service (Service Business Models (for capturing services provided)) is at http://github.com/walterjwhite/model.business.service/
model.business.restaurant (Restaurant Business Models (for capturing billing information)) is at http://github.com/walterjwhite/model.business.restaurant/
model.news (News models) is at http://github.com/walterjwhite/model.news/
model.project (Project models (for tracking projects, work, tasks, etc.)) is at http://github.com/walterjwhite/model.project/
model.meal (Meal models) is at http://github.com/walterjwhite/model.meal/
model.gaming.rFactor (rFactor game models) is at http://github.com/walterjwhite/model.gaming.rFactor/
model.question-and-answer (Question and Answer models) is at http://github.com/walterjwhite/model.question-and-answer/
model.feedback (Feedback models) is at http://github.com/walterjwhite/model.feedback/
linuxtag2010_javaEE6 (Example code of Java EE 6 tutorial at Linuxtag 2010.) is at http://github.com/walterjwhite/linuxtag2010_javaEE6/
deprecated.navigation.web.automation (Navigation Web automation module) is at http://github.com/walterjwhite/deprecated.navigation.web.automation/
model.career (Career models / entities) is at http://github.com/walterjwhite/model.career/
model.clothing (Clothing models / entities) is at http://github.com/walterjwhite/model.clothing/
deprecated.model.crawler (deprecated, these projects will be split up (reference purposes only)) is at http://github.com/walterjwhite/deprecated.model.crawler/
resume (My Resume) is at http://github.com/walterjwhite/resume/
business-cards (Business Cards) is at http://github.com/walterjwhite/business-cards/

The answer I was waiting for.

I decided to give Ruby a try and it is okay. I like how it is compact, but it isn't pretty looking :(.

This works:

#!/usr/bin/env ruby
require "yaml"
require "open-uri"

time = Time.new
backupDirectory = "/storage/backups/github.com/#{time.year}.#{time.month}.#{time.day}"
username = "walterjwhite"

#repositories =
# .map{|r| %Q[#{r[:name]}] }

#FileUtils.mkdir_p #{backupDirectory}

YAML.load(open("http://github.com/api/v2/yaml/repos/show/#{username}"))['repositories'].map{|repository|

    puts "found repository: #{repository[:name]} ... downloading ..."
    #exec
    system "git clone git@github.com:#{username}/#{repository[:name]}.git #{backupDirectory}/#{repository[:name]}"
}

Walter

Now that the v2 API used in the accepted answer no longer works, it's time for an update that uses Github API v3.

You can get the list of repositories in JSON format with

curl -i https://api.github.com/users/username/repos

Watch out for pagination! By default the results are paginated to 30 items. If you have more repositories than fit into a single page, you'll get a Link HTTP response header with links to the other pages (with rel=next/last/first/prev). You can also ask for a larger page size (up to 100):

curl -i https://api.github.com/users/username/repos?per_page=100

A full backup script (assuming you have 100 or fewer repositories) would look something like this:

#!/usr/bin/python
import os
import json
import urllib
import subprocess

username = 'username'  # specify your github username
os.chdir(os.expanduser('~/github'))  # location for your backups, must exist

url = 'https://api.github.com/users/%s/repos?per_page=100' % username
for repo in json.load(urllib.urlopen(url)):
    print "+", repo['full_name']
    if os.path.exists(repo['name']):
        subprocess.call(['git', 'pull'], cwd=repo['name'])
    else:
        subprocess.call(['git', 'clone', repo['git_url']])