etl - Talend internal variables -

i doing data migration project in talend , 1 of tasks need process big table many columns , map (old) data different value new model.
i have unique mapping table 3 columns.

example:

 column name | value old | value new       "col 1"         1           3       "col 1"         3           2       "col 2"         10          7         etc 

that way can refer using "column name" values related column need map.

using old value lookup in mapping table using "column name": when old value matches, return new value.

i manually each column, there hundreds mean @ least 1 hash repeated every column.

i stumped how more , once every possible column in original table.

right doing huge tmap , using lots of lookups: 1 each column.

any ideas appreciated.

-

some ideas had:
1) there way know name of link joining 2 components? reuse same connection , filter on tmap more easily

i solve using tmemorizerows, saving whole mapping table memory using tjavaflex , reading them using routine.

what did was, memorize whole table, had 1 table, 3 columns, first indicating type of mapping had do, second being original code, , third 1 being replacement code.

once memorized data, used tjavaflex create 3 lists, 1 each column , move lists global variables map.

then created routine receives 3 lists plus 2 codes, first 1 mapping name (first column filter) , second 1 original code (second column filter). using both pinpoint position of replacement code, , return it.

finally, when reading main flow of data, using tmap, created variable each distinct value of first column (that is, 1 each type of mapping) , called routine developed using oriinal code + mapping name.

the code used in tmemorizerows was:

start code

java.util.set<string> ilista; java.util.list<string> llov=new java.util.arraylist<string>(); java.util.list<string> ls6=new java.util.arraylist<string>(); java.util.list<string> ls8=new java.util.arraylist<string>(); 

main code

llov.add(lov_tmemorizerows_1[icount]); ls6.add(s6_name_tmemorizerows_1[icount]); ls8.add(s8_name_tmemorizerows_1[icount]); 

end code

globalmap.put("llov",llov); globalmap.put("ls6",ls6); globalmap.put("ls8",ls8); 

routine code

/*  * toma los valores generados de la lov y matchea con s8, requiere la ejecucion del job  * genericos\generadorlov  *   * {talendtypes} string | string  *   * {category} migraciondatos  *   * {param} string(entrada.lov) identifica el tipo de lov  *   * {param} string(entrada.s6_name) indica el valor del campo en siebel 6   *   * {param} ((java.util.list<string>) globalmap.get("llov"))  *   * {param} ((java.util.list<string>) globalmap.get("ls6"))  *   * {param} ((java.util.list<string>) globalmap.get("ls8"))    *   */    public static string calculalov(string campolov, string campos6, java.util.list<string> listalov, java.util.list<string> listas6,java.util.list<string> listas8   ) {        /*         * java.util.list<string> listalov = ( java.util.list<string>) globalmap.get("llov");         * java.util.list<string> listas6 = ( java.util.list<string>) globalmap.get("ls6");         * java.util.list<string> listas8 = ( java.util.list<string>) globalmap.get("ls8");         */         string c1 = campolov;        string c2 = campos6;         int posicionc1 = listalov.indexof(c1);         // encontró el lov        if(posicionc1 >= 0) {          java.util.list<string> listas6_iterada = new java.util.arraylist<string>();          // genera la lista intermedia con los valores         (int contador = posicionc1; contador < listalov.size() ; contador++) {             listas6_iterada.add(listas6.get(contador));             if(!listalov.get(contador).tostring().equals(c1)) {break;}         }          int posicionc2 = listas6_iterada.indexof(c2);         if(posicionc2 >= 0) {              int posicionfinal = posicionc1 + posicionc2;              return listas8.get(posicionfinal);         } else {             return "";         }        } else {         return "";        }    }