Hi,
I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.
I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).
Manual: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt
Tutorials: https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree
We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:
* create dates, dseries, and reporting repositories, with master branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.
* in the dates, dseries and reporting repositories, include dynare.git as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.
* when these files are updated in Dynare, we need to do a: git subtree pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.
* on the dynare branch of each of these repositories, we keep only the code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.
I think this solves all of the problems we had been talking about. Is this ok with everyone?
Best, Houtan
On 02/11/2015 02:56 PM, Stéphane Adjemian wrote:
Hi,
I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments
(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …
(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.
(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).
If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:
(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).
(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.
I favor the first solution (because when we distribute code we do not distribute the entire history)
Best, Stéphane.
Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit :
Let see what Stephane says.
Houtan writes:
This sounds good to me.
Can you create the repository in Github? I'll take care of moving everything over and making sure it works.
Best, Houtan
On 02/09/2015 04:25 PM, Michel Juillard wrote:
After some thinking, my proposal is in fact to maintain a single GitHub repository called cepremap_toolbox in lack of a better name, that would contain, dates, dseries, reporting and utilities used both in Dynare and in these projects.
Then to stop the separate GitHub repositories dates and deseries
The idea that for the moment it is not a huge imposition for somebody who wants to use only our dates to download everything. If and when cepremap_toolbox becomes too large we will think about a better packaging.
Best
Michel
Stéphane Adjemian writes:
Hi,
I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to:
(1) update the dates repository (2) update submodule dates in cepremap-tools repository (3) update submodule cepremap-tools in dynare repository
I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries.
Best, Stéphane.
Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit :
I'm in favor of having dates and dseries and, maybe reporting, work independently from Dynare. In a first stage, I don't see it as a big drawback to have to download separately dates and dseries. At a later stage, we may think about have a more user friendly distribution. The more delicate issue for which I don't have a simple solution is those files upon which both dseries, reporting and Dynare rely upon: check_file_extension.m load_csv_file_data.m name2tex.m iscellofchar.m etc. The logic would be to have yet another submodule like cepremap_utilities, but it seems a little farfetched. I'm thinking as I'm writing: maybe a better approach is to start with a single module cepremap_toolbox (or any other name showing its independance from Dynare) with dates, dseries, reporting and common utilities and only think about splitting it up when one part is growing too much.
Best
Michel
Houtan writes:
> Hi.... > > On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> Hi, >> >> Answers below... >> >> Le 26/01/2015 10:40, Houtan a écrit : >>> Hello, >>> >>> If I should have any spare time this week, I wanted to work on >>> https://github.com/DynareTeam/dynare/issues/811. >>> >>> Looking into this has brought up several questions. If I >>> understand correctly, the idea of moving dates and dseries into >>> their own repositories was that they can be used as standalone >>> programs. As such, we want to permit people who can benefit from >>> this code to use it directly instead of having to download Dynare >>> as a whole. In essence, we're treating them as libraries, which I >>> think is a good idea. >> Initially the idea was to make these codes available without >> downloading the whole Dynare. But I did not intend to produce >> standalone programs. For me it's not a problem if the dseries package >> requires another package (dates package in this case). I do not like >> the idea of having dates as a submodule of dseries, or dates and >> dseries as submodules of reporting, ... I do not see any advantage >> here, only more work. > The advantage is to simplify the process for the end user, having > something that requires minimal setup. Anyway, as it stands now (though, > as you point out, we could make a script that does this), the end user > still needs to download dynare in order to have access to some of the > files that these submodules depend on. E.g.: > check_file_extension.m > load_csv_file_data.m > name2tex.m > iscellofchar.m > etc. > > In fact, if the only advantage is to save the user from downloading the > 12mb of dynare source code, and if you don't intend to make these > submodules work independently, I don't see the advantage of spinning > them off into submodules in the first place. It actually just serves to > make our code more convoluted. I think that would be worth it if we are > willing to put in the work to make the products (dates, dseries, and > reporting) completely independent. But, if not, it just doesn't seem to > make sense. > > To be clear, I don't think it's much work to make these work independently. > > We would have to include the files above in a "utilities" submodule, > that would include general matlab functions that are used in various > parts of dynare (the files that you have already placed in > matlab/utilities/general in addition to some others). Then, as I > specified before, include dates as a submodule in dseries, and dates and > dseries as submodules of reporting. It doesn't seem like too much work > at all. What is the problem with this? > > Best, > Houtan > >>> But, as it's setup now, dseries is not a standalone program. In >>> order to make it a standalone program, we'd need to add dates as a >>> submodule of this program and probably have a configuration script >>> that adds the dates repository to the Matlab/Octave path. At least, >>> that's how I'd like to set up reporting: add dates and dseries as >>> submodules to reporting so that someone could clone the reporting >>> repository, initialize its submodules and use it as a standalone >>> program. >> We just need to explain that if one wants to use the reporting module, >> then the dates and dseries modules must be in the Matlab/Octave's >> path. We can also provide a script to install these modules (but we do >> not need git to do that). >> >>> Doing so would not present a problem to Dynare users as Dynare >>> could still include a dates submodule, a dseries submodule that >>> contains a dates submodule, and a reporting submodule that contains >>> both a dseries and dates submodule. As long as the git clone or the >>> git submodule init are not issued with the --recursive switch, the >>> nested submodules will not be brought in and hence the code will >>> not be duplicated. >>> >>> Furthermore, I think we should discuss the location of submodules >>> within Dynare. Currently, they're in several places: >>> >>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>> >>> This is confusing. Though I don't have a strong opinion as to the >>> logic we should impose on the submodules we include, I believe >>> there should be a logic. Should all external submodules be included >>> in contrib? should all internal matlab submodules be included in >>> matlab/modules? Should all matlab tests be included in >>> tests/modules (matlab/utilities/tests has to be changed, in my >>> opinion)? It would be nice if we could make a decision about the >>> organization of the submodules within Dynare as a group so that, >>> going forward, we have a coherent structure that does not evolve >>> organically but rather in an organized and logical way. >> I plan to put the tests routines under matlab/modules. I think that we >> should have only two locations. The logic being that some codes are >> supported/produced by us, in modules folder for instance, and other >> codes are not produced/supported by us (we do not ensure that these >> codes will work in future versions of Dynare), in a contribs folder. >> >> Best, >> Stéphane. >> >> >> >>> Best, Houtan _______________________________________________ Dev >>> mailing list Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> - -- >> Stéphane Adjemian >> Université du Maine, Dynare Team >> -----BEGIN PGP SIGNATURE----- >> >> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >> =2Aq0 >> -----END PGP SIGNATURE----- >> _______________________________________________ >> Dev mailing list >> Dev@dynare.org >> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >> > _______________________________________________ > Dev mailing list > Dev@dynare.org
> https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
-- Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev