Dear George,
Of course - this approach is for slightly higher level and more complex modules such as estimate or few other mexFunctions which then can pass the once retrieved individual parameters around, e.g. to the (lower level) functions they are calling in a loop (or not), so to reduce the overhead of the Matlab-2-C++ communication.
Another aim is also to reduce unnecessary copying of whole structures such as options_.
One current assumption is that the names of the main parameters in the Dynare structures are mutually unique across the spectrum - can we guarantee that in future?
I'm not sure that the assumption is true currently. We have started using two level options names (in fact a Matlab structure in a structure). Maybe you should mimic that as well.
Cheers
Michel
Best regards
George
----- Original Message ----- From: "Michel Juillard" michel.juillard@ens.fr To: "List for Dynare developers" dev@dynare.org Sent: Monday, September 14, 2009 12:36 PM Subject: Re: [DynareDev] C++ DsgeLikelihood and Dynare Parameters
Thanks George, this will be very useful at the higher level. However, for functions that are repeatedly called in a loop, it should be faster to pass the (few) parameters as arguments rather than searching the map inside a loop.
All the best,
Michel
G. Perendia wrote:
Hi all
I have been working on a fundamental approach to the further below
outlined
(see enclosed email below) issue of dynare structures, either global or local, being available inside C++ modules and have developed prototypes
of
two classes that provide for integration of the parameters inside one
class
and for their access and modification. (The location ands scope of the original Dynare param structure need not be global but can be also local
to
the calling Dynare matlab modules.)
They provide for:
- a generic abstract class GeneralParams as interface to parameters
repository from C++ :
class GeneralParams { map <string, int> params; const char *structName; public: GeneralParams(); GeneralParams( char *sBase, char * parStructName); virtual ~GeneralParams(); virtual void * getField(char * field)=0; virtual char * getCharField(char * field)=0; //returns General Matrix from sylv library virtual GeneralMatrix * getMatrixField(char * field)=0; // overloaded pramater update "set" methods: virtual void setField(char * field, char *newVal)=0; virtual void setField(char * field, double val)=0; virtual void setField(char * field, GeneralMatrix& val)=0; };
another dealing with the specifics of Matlab and Dynare:
use mexGetVariablePtr() to locate pointer to the structure rather than
copy whole structure locally into C++ memory space and makes them
available
read only-
- concatenates Dynare structure individual elements' names into a single
map
mapping their names to the original structure pointers so that they can
be
located and retrieved individually directly form the Matlab memory space when if needed.
- Allows for structures' elements to be updated and propagated back
inside
Matlab (locally or globally). This approach then uses mexGetVariable()
to
copy structures into C++ pace only when one of the structure values
needs
to be updated, and pus it back.
The generic interface abstract class leaves thus to the Dynare Matlab specific implementation to deal with any changes in Dynare structures
and
their locations that may occur in meantime.
The system still requires testing and it is in a rough prototype stage,
and
it is also, of course, open to further suggestions and modifications and
I
would like your suggestions before committing its design to the WikiPage
and
its code to thorough testing.
Best regards
George artilogica@btconnect.com
----- Original Message ----- From: "Michel Juillard" michel.juillard@ens.fr To: "G. Perendia" george@perendia.orangehome.co.uk; "List for Dynare developers" dev@dynare.org Sent: Thursday, August 27, 2009 5:39 PM Subject: Re: C++ DsgeLikelihood
Dear George,
thanks for the message. The issue needs to be discussed carefully.
- We are trying to get rid of most global variables and to pass oo_,
options_ and M_ as arguments. There is no good reason, why this has not been done yet in DsgeLikelihood.m 2) Besides a cleaner more robust code, we learned that accessing global variables in Matlab is expensive, and, most likely it is also true for Mex library 3) DsgeLikelihood++ is not primarily meant for calling from Matlab, but from a C++ version of dynare_estimation_1.m 4) There are only two of these variables that are changed in
DsgeLikelihood:
- M_.params, but a local version of modified params vector should
probably be an argument to dynare_resolve
- options_.kalman_algo that passes back the choice of Kalman algorithm.
We can probably find another mechanism to pass back this information to Matlab. 5) My preference would be to have one C++ class that contains all the parameters from M_, options_ and bayestopt_ (the latter needs to be rethought), needed by the C++ estimation functions. This class should
be
initialized at the instanciation of the top class (here dynare_estimation++) from M_ and options_ pointers passed as arguments 6) One needs to think about a similar mechanism for output variables. 7) I have three problems with the current structures that we use *M_ contains parameters describing the model (OK) *oo_ contains output. We may want to distinguish output that is only made available to the user and intermediary output that is meant to be used by other routines such as structure dr. It could make accessing these intermediary results faster
- options_ contains both real options_ and information on the current
state of Dynare computations (such as kalman_algo here), These should probably be separated
- bayestopt_ (and estim_params) is currently a mess and should be
reconsidered. It is mainly meant to contained parameters describing the priors that are deduced from user input passed by the preprocessor in estim_params 8) In summary, for now, in DsgeLikelihood++, you should simply refer to a C++ version of these objects that we will build later.
Best
Michel
G. Perendia wrote:
Dear Michel
Regarding design of the interface with Matlab (and Octave?), there are
many
variables from global structures that need to be referred to from
various
parts of estimation (and some updated too).
Is it ok if I make DsgeLikelihood++ system refer to options_, oo_ and
other
large structures (and some other variable) using mex functions mexGetVariable (and, if needed, mexPutVariable for updates) rather
than
trying to pass whole structures at the call time and creating C++
instances
of all or (large) subsets of their data.
e.g. const mxArray *mexExt = mexGetVariable("base", "mexext"); or const mxArray *M_ = mexGetVariable("caller", "M_");
That will reduce number of variables that need to be passed on call
and
writing code on their breaking down and passing around from
mexFunction
to
the other modules.
I can optimise access to read-once only to reduce DLL-Matlab traffic
too
using a permanent class on the lines of:
class MexStruct { mxArray* mexStruct; const char *base; // one of: base, caller, global const char *structName; MexStruct( char *sBase, char * mexStructName) : base(sBase), structName(mexStructName) { mexStruct=mexGetVariable(base, structName); } mxArray * getMexStructField(char * field) { return mxGetField(mexStruct, 0, field); } void putMexStruct(){mexPutVariable(base, structName, mexStruct);} void setMexStructField(char * field, mxArray *newVal) { mxArray *fp = mxGetField(mexStruct, 0, field); mxDestroy(fp) mxSetField(mexStruct,0,field, newVal); } }
Some global variables in those structures also need to be set so the
above
class' methods can pass back updated global variables too at the end
of
the
DsgaLikelihood. Also, as another advantage of this model, they can be
passed
back even in real time when if that may be needed for parallelisation.
Best regards
George Mob. +44(0)7951415480 artilogica@btconnect.com
----- Original Message ----- From: "Michel Juillard" michel.juillard@ens.fr To: "G. Perendia" george@perendia.orangehome.co.uk Sent: Monday, August 24, 2009 11:15 AM Subject: Re: C++ DsgeLikelihood
Hi George,
> Dear Michel > > 1) Re email below, due to problem of code for calling > > >
<model>dynamic.dll,
> needed by estimation but being in the same file with mexFunction in > k_order_perturbation.cpp > > Alternatives: > a) I can use (a different) literal, e.g. > KORDERPERT_MEX_FILE > and compile k_order_perturbation.cpp for k_order_perturbation.dll > >
using
> -DKORDERPERT_MEX_FILE > and for DsgeLIkelihood/estimation without, - probably the easiest > > >
solution
> to implement and maintain until situation may change, - or, > > b) I can split k_order_perturbation.cpp in two - > > >
k_order_perturbation.cpp
> with mexFunction and another file for calling <model>dynamic.dll >
which
>
will
> be placed into korderpertlib.a and linked to >
DsgeLikelihood/estimation
>
too -
> probably the optimal solution. or, > > c) copy k_order_perturbation.cpp code for calling > > >
<model>dynamic.dll.into
> another file and keep it under DsgeLikelihood/estimation and have > DsgeLikelihood refer to k-orderpertlib for the rest - this may > >
duplicate
> effort on maintaining the dynamic.dll call management code but it is >
a
>
clear
> cut. > > > > > I prefer 1.b) too
> 2) Shall I treat C++ DsgeLIkelihood > a) as an extension of KF or KORDERPERT and put its code in one of > >
those
2
> mex/source directories, or > > b) create a new one, e.g. estimation, with its makefiile referring >
to
>
the
> code and objects in the other two? > > > I suggest 1.b and to treat DsgeLIkelihood as a subdirectory of >
kalman
> directory which will refer to korderpertlib and kalman src. > > > > > I prefer 2.b, because DsgeLikelihood calls Kalman code and it is
weird
to have it in a subdirectory of kalman
All the best,
Michel
> Please let me know if that is OK with you. > > Best regards > > George > > ----- Original Message ----- > From: "G. Perendia" george@perendia.orangehome.co.uk > To: "Michel Juillard" michel.juillard@ens.fr > Sent: Friday, August 21, 2009 3:50 PM > Subject: Re: C++ DsgeLikelihood > > > > > > >> Dear Michel >> >> In k-order-perturbation makefile someone(you?) defined literal >> -DMATLAB_MEX_FILE >> for several cpp files, some of which do not contain mexFunction, >>
e.g.
>> k_order_test_main.o: k_order_test_main.cpp >> gcc -DMATLAB_MEX_FILE ... >> >> k_ord_dynare.o: k_ord_dynare.cpp k_ord_dynare.h >> gcc -DMATLAB_MEX_FILE .... >> >> k_order_perturbation.o: k_order_perturbation.cpp >> >>
k_order_perturbation.h
>> gcc -DMATLAB_MEX_FILE .... >> >> nlsolve.o: $(DYNAREPPDIR)/src/nlsolve.cpp >> gcc -DMATLAB_MEX_FILE .... >> >> whereas only k_order_perturbation.cpp contains mexFunction(). >> >> What is that switch/literal used for? >> >> I wandered if it is OK to use that literal and set a switch in >> k_order_perturbation.cpp to exclude mexFunction from compilation >>
when
>>
not
>> defined so that the file can be used with other mexFunctions such >>
as
>>
the
> one > > > > >> for DsgeLikelihood., e.g. >> >> #ifdef MATLAB_MEX_FILE // exclude mexFunction for other >>
applications
>> extern "C" { >> void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const >> >> >>
mxArray
>> *prhs[]) >> { >> ... here goes mexFunction .: >> } >> }; // end of extern C >> #endif // ifdef MATLAB_MEX_FILE >> >> Oterwise I can set another literal. >> >> Best regards >> >> George >> >> ----- Original Message ----- >> From: "Michel Juillard" michel.juillard@ens.fr >> To: "G. Perendia" george@perendia.orangehome.co.uk >> Sent: Tuesday, August 18, 2009 12:36 PM >> Subject: Re: C++ DsgeLikelihood >> >> >> >> >> >> >>> Hi George, >>> >>> currently there are 3 different cases: >>> >>> 1) the steady state is not affected by estimation. This is the >>>
case
>>>
that
>>> we are implementing now >>> 2) A Matlab function is provided by the user to compute the steady >>> >>> >>>
state
>>> as a function of model's parameters. >>> The implementation is cumbersome for the user and should become a >>> feature of the preprocessor. As long as the definitive version is >>> >>>
not
>>> done, there is no point in losing time writing a C interface for >>> >>>
that
>>> 3) The steady state must be solved for by calling a nonlinear >>> >>>
equation
>>> solver. We will need to implement this as well, but it is a lower >>> priority. I would like to have the entire estimation chain in C++ >>> working first with just a constant steady state. >>> >>> All the best, >>> >>> Michel >>> >>> >>> G. Perendia wrote: >>> >>> >>> >>> >>>> Dear Michel >>>> >>>> Regarding steady state, can you remind me why we can not use ss >>>> >>>> >>>> >>>> >> calculation >> >> >> >> >>>> inside the k-order-perturb ? >>>> Can we than instead call Dynare Matlab steady state calculation, >>>> >>>> >>>> >>>> >> currently >> >> >> >> >>>> called from resol.m, from estimate_1.m just before calling the >>>> >>>>
new
> C++ > > > > >>>> DsgeLikelihood (or from a new Matlab shell DsgeLikelihood_m.m >>>>
that
is
>> then >> >> >> >> >>>> calling C++ DsgeLikelihood) where new C++ resol will not call ss >>>> calculation as the Matlab one does? >>>> >>>> Best regards >>>> >>>> George >>>> >>>> ----- Original Message ----- >>>> From: "Michel Juillard" michel.juillard@ens.fr >>>> To: "G. Perendia" george@perendia.orangehome.co.uk >>>> Sent: Monday, August 17, 2009 8:37 PM >>>> Subject: Re: C++ DsgeLikelihood >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> Hi George, >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Should I suppose that the C++ DsgeLikelihood would also include >>>>>> >>>>>> >>>>>>
C++
>>>>>> versions of: >>>>>> resol.m >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> yes, this mainly calls first order solution in dr1.m >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> kalman_transition_matrix.m >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> yes >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> and >>>>>> the 1st order part of the k-ord_pert version of dr1.m that will >>>>>> >>>>>>
be
>>>>>> >>>> calling >>>> >>>> >>>> >>>> >>>> >>>>>> k-order_perturbation for 1st order only, either as a DLL (as >>>>>>
it
>>>>>> >>>>>> > does > > > > >>>> now) >>>> >>>> >>>> >>>> >>>> >>>>>> or incorporating it statically? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> first order solution should be linked statically >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> May I then also assume that >>>>>> >>>>>> a) set_state_space.m and the kstate preamble will in future be >>>>>> >>>>>> >>>>>> >>>>>> >> performed >> >> >> >> >>>>>> before calling DsgeLikelihood, and not repeatedly within dr1 >>>>>>
and
>>>>>>
its
>>>> future >>>> >>>> >>>> >>>> >>>> >>>>>> C++ version - as I remember has already been planned? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> yes, we will have to implement such a mechanism >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> b) Ramsey will not be part of it for now >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> correct >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> c) I need to check but I think I may not need to code >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> transition_matrix.m >>>> >>>> >>>> >>>> >>>> >>>>>> and use the info 3/4 test currently in dr1: >>>>>> if nba > nyf info(1) = 3; >>>>>> elseif nba < nyf info(1) = 4; >>>>>> as a similar test already exist in k-ord-pert. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> probably correct >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> d) Also, we seem to need a new algorithm to handle exogenous >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> deterministic >>>> >>>> >>>> >>>> >>>> >>>>>> variables i.e. when M_.exo_det_nbr > 0 ? They are not currently >>>>>> >>>>>> >>>>>> >>>>>> >> handled >> >> >> >> >>>> for >>>> >>>> >>>> >>>> >>>> >>>>>> k-order perturbation. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> no, there is no estimation with exogenous deterministic >>>>>
variables
>>>>>
>>>>> Best >>>>> >>>>> Michel >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Best regards >>>>>> >>>>>> George >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Michel Juillard" michel.juillard@ens.fr >>>>>> To: "G. Perendia" george@perendia.orangehome.co.uk; "Joe >>>>>> >>>>>> >>>>>>
Pearlman"
>>>>>> j.pearlman@londonmet.ac.uk >>>>>> Sent: Friday, July 31, 2009 8:46 AM >>>>>> Subject: Re: lyapunov kalman filter options >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Thanks George, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> .... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> For the next steps, I think that you have now all elements to >>>>>>>
co
de
> a > > > > >>>> C++ >>>> >>>> >>>> >>>> >>>> >>>>>>> version of DsgeLikelihood.m Of course you should restrict it >>>>>>>
to
>>>>>>>
the
>>>> case >>>> >>>> >>>> >>>> >>>> >>>>>>> of stationary models. >>>>>>> >>>>>>> As we haven't solved yet the issue around computing the steady >>>>>>> >>>>>>> >>>>>>> >>>>>>> > state, > > > > >>>>>>> for the time being, limit yourself to the case where the >>>>>>>
steady
>>>>>>> >>>>>>> > state > > > > >>>>>>> doesn't change throughout estimation. >>>>>>> >>>>>>> Best >>>>>>> >>>>>>> Michel >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>> > >
Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org http://www.dynare.org/cgi-bin/mailman/listinfo/dev