Home Forums Numerical Uncertainty Tools Uncertainty estimation formula used with the 2017 version of the tool

Viewing 12 posts - 1 through 12 (of 12 total)
  • Author
    Posts
  • Andrea Mikelic
    Participant
    Post count: 5

    Dear fellow ReFRESCO user,

    I am working on a verification and validation study. I use to use the previous version of the tool 2016 that came with an excel sheet. It was quite handy to get the uncertainty of all the grid and not only the finest. The new version doesn’t have one, as far as I know.
    So I recomputed the uncertainties using the fit values and the formula from the paper of Luis Eca and Martin Hoekstra from 2014 given in the doc and I obtain a different result then given by the tool. I obtained from the tool 9.2% and from my estimation 3.5% as a maximum. As my data are not monotonic I put myself in the most conservative position.
    I also tried the procedure from the 2010 ECCOMAS paper but I also get different results.

    Which formulas are used by the tool to compute uncertainties ?

    Pierre Crepier
    Keymaster
    Post count: 14

    Hi Andrea,

    I’m afraid I can’t answer your question without asking for further details here at MARIN, but I know that it is possible to get the uncertainties of other points than the finest one (step 1) by changing slightly the input format of the data.
    I worked only with the version of 2016 but I expect the data format of the version of 2017 to be the same.

    Could you post a sample of your data here (the file containing the forces values, I guess it is forces values), then I might be able to give you hints on how to get this.

    If you can’t give the data, the input format works like that:
    line 1: header of the columns
    line 2: number of points to use in the fit
    line 3 and 4: give the order in which the data is ordered: if the first column of line 4 is larger than that of line 3: increasing order. decreasing for the opposite case

    The uncertainty will be given for the last line of your input file. That means that you do not have to have the finest or coarsest grid (depending on the sorting order of your data) as last line, but you can put one of the intermediate grids.

    Is this helping you ?

    refrescoadmin
    Keymaster
    Post count: 11

    Andrea,

    Dont try to reproduce the formulas of the papers; it will not work. See the manual and the documentation that comes with the V&V package, where all is explained. However, the new version 2016/2017 have lots of iterative schemes for robustness statistics that I dont think you can do it easily with Excel (perhaps with Python). But the idea of spreading these tools is precisely not to have people trying to reproduce them :-).

    About the uncertainty for all grids/time-steps/calculations I thought uncertainty/2017-rc2 (last version http://www.refresco.org/verification-validation/utilitiesvv-tools/) did that by default (I may be wrong though…)

    Andrea Mikelic
    Participant
    Post count: 5

    Thanks Pierre I will give it a try. I did not know that the last line was always the one where the uncertainty was given, (even though looking back at the picture it looks obvious).

    Guilherme, I will be very happy not to have to recode anything indeed 🙂
    I was trying to get back the uncertainty using the values form the given fit. Is the fit that cannot be reproduce or the uncertainty computation based on it?
    In the documentation of the 2017 version there is no details about how things are computed as the guide_numuncertainty_module.pdf is not present anymore. That guide that was in the 2016 package, indeed gives details on how the fit and the coefficients are obtained but refers to the paper of 2014 for the uncertainty computation.

    Maybe in the future in the latex tables we could have the uncertainty for all the grids or an equivalent kind of excel sheet than with the 2016? I liked that sheet a lot it helps make things clearer. I cannot use it with the 2017 report.out at least not using the refinement ratio as input.

    Pierre Crepier
    Keymaster
    Post count: 14

    Andrea,

    I am also now busy using the tool, so I asked how I could get the uncertainties for all the points and … it’s not possible yet with 2017-rc2 …
    This comes in the next release, hopefully somewhere in January next year!

    So in the meantime, you can still use the trick i gave earlier: putting the line for which you want the uncertainty at the last line.

    Andrea Mikelic
    Participant
    Post count: 5

    Hi Pierre,

    I gave it a try and I managed in most cases to get the uncertainty value and the according graph. In some cases however the answer changes. it seems not to take the same points for the fit depending of which uncertainties are considered and also of the data.
    By the way, in the 2017 version in steady mode it does not accepts having the coarsest grid first. The following message appears:

    ERROR: For steady mode, UncertaintySolution must indicate the finest grid solution

    Any chances to have other output then tecplot for the unsteady modes in the new version?

    Carlo Negrato
    Participant
    Post count: 6

    By the way, in the 2017 version in steady mode it does not accepts having the coarsest grid first. The following message appears:

    ERROR: For steady mode, UncertaintySolution must indicate the finest grid solution

    I quote Andrea on this. Same situation for me, steady case, tool 2017-rc2. Not able to get the uncertainty for a grid other than the finest.

    So in the meantime, you can still use the trick i gave earlier: putting the line for which you want the uncertainty at the last line.

    Pierre, this also did not work for me.
    I follow this thread and wait for new release to check it out.
    Carlo

    Pierre Crepier
    Keymaster
    Post count: 14

    Carlo,

    Could you send or paste your input files in a post so that I could pass them through the version.
    I only used the version of 2017 with steady data, but I was able to get the uncertainty for any point I wanted …
    So I am curious to see what I different from you.

    Pierre

    Carlo Negrato
    Participant
    Post count: 6

    ——————————————–
    NCells Var1 # This line contains variable names
    0 10 # This line contains the number of results to use for the fit
    0 32.55 # Optional line with experimental results
    1568000 36.7
    2244000 36.4
    2945250 35.71
    3528000 36.3
    3920000 31.7
    5610000 35.7
    6174000 36.3
    6272000 34.8
    8820000 34.78
    15680000 32.5
    ——————————————————–
    [Numerical_uncertainty]
    CaseName = carloTest
    DatafileName = var1_gridref.dat
    OutputFormat = PNG ; PS/TEX/PNG/WIN/TEC
    IsUnsteady = No
    GridStepsizeMethod = 3
    TimeStepsizeMethod = -1
    UncertaintySolution = 0
    Showp = Yes
    —————————————————-

    Pierre: for completeness:
    – the ‘UncertaintySolution = 0’ is the only value accepted for the data in ascending order
    – actually, if I put another grid at last, I do have the error bar for a different grid as you said. BUT the fit is different which, to my understanding of the method, is not the expected behavior.
    Very possible that I am missing something here…. Thanks for getting back ,
    Carlo

    Pierre Crepier
    Keymaster
    Post count: 14

    Ok, so your second remark is what I meant. For the data I had, I did not saw differences in the fits (or maybe did not wanted to see them :)), so I was able to get the uncertainties for my other points.

    I am not sure when they will release the next version, but the possibility to have all uncertainties is definitely there!

    Pierre

    refrescoadmin
    Keymaster
    Post count: 11

    Pierre,

    There is a bug in v2017, as Serge explained in an email. You can try both versions and you will see the difference on the results.
    That is why I asked if I could deploy the new v2018 version…
    Perhaps I can give that version to Carlo for him to test it?

    Guilherme V.

    refrescoadmin
    Keymaster
    Post count: 11

    Guys,

    New version 2018 of the V&V tools have been published here: http://www.refresco.org/verification-validation/utilitiesvv-tools/
    Could you please test it and give us feedback?

    Thanks,
    Guilherme & Serge

Viewing 12 posts - 1 through 12 (of 12 total)
  • You must be logged in to reply to this topic.