Roads ? - where we're going, we don't need roads ...
FBI internal investigations...hidden virtual columns in 8i. | ORA600
ORA600 content Content RSS
Oracle ORA600 News RRSS Oracle News RSS
ORA600 blog Blog RSS
ORA600 blog Blog Atom

User login

FBI internal investigations...hidden virtual columns in 8i.

I was called in last week for a performance issue at a customer site.
Let's set scene a bit …

They had moved their main production OLTP database from AIX to Windows and had been struggling with performance for a couple of weeks. Hardware-wise, they went from expensive but old Bull hardware to a Dell box (with 8 cores). And they went from 4Gb to 16Gb RAM. So you would expect the database would fly on this new hardware.
Now - this database was/is pulling around a 1000 concurrent users…. The first thing they ran into was the process' addressable memory space on 32bit windows, being 2Gb. This of course led to a lot of ORA-04030 and angry users.  Actually, Anjo recently made a blogentry about it.

So actions were made to optimize Oracle for windows, which basically consists of putting the buffercache above the 2Gb limit - anyway metalink note 231159.1 explains all of this, but still, as this is an Oracle 8.1.7.4 database handling a thousand users, all other memory (remember sort_area_size, hash_area_size etc) related parameters were set to a minimum to avoid ORA-4030's.
The result of this memory balancing operation was, that performance was below average. And I was send in to have a look at it. I must admit - I hadn't tuned a whole environment for a while, but I had done a lot of 8.0 and 7.3 migrations to 8i back in the day, and knew pretty much what the problems of the 8i CBO were. Oh - why are the still running 8i … well the database application is to be decommissioned this year and so they didn't bother to move up to the latest Oracle version. The fact that they wanted to invest in getting performance up, proves power users were about to lynch the IT manager Tongue out
Anyway - 5minutes into the meeting with IT, a second consultant from a rivaling company walks in. That's how bad performance was. I don't mind competition - in fact I didn't see it as competition. The guy was more than reasonable and during the day we acted as each-others soundboard.

The first thing I noticed were the bad response times on the disks and the large amount of physical IO the database was doing. The database running on a Dell SAN…. I didn't even know Dell produced SANs !!! :-p But I knew one thing - this SAN was sh*te. It used 8 disks in RAID5 (coming from 4 disks) and I reckon the write cache must have been quite small. In fact at first I thought the SAN cache must have been disabled - something I had encountered multiple times when the battery of the cache dies, the cache on the SAN is disabled to avoid corruption in case of power-failure.
Clearly, this DB was doing too much IO for the SAN… way past it's critical point in the hockey stick chart see also Anjo's paper on this  - oh and we couldn't increase any memory settings within the DB of course.


So the only option left, was hunting down individual sql statements and tune them, so the overall IO would decrease.  The impact of sql tuning can be *huge* - especially with a 1000 oltp users. Just imaging a sql that does 5000 physical IO's  that is run by these 1000 users every minute (5x10^6 IO/min) - if we can tune it to use only 20 physical IO's (20x10^3 IO/min)  than that would have a dramatic effect on overall performance.


I noticed that there were a lot of full tablescans … that's not necessarily bad thing, but sometimes it just is ! From the old days, I knew that Oracle 8i's CBO (with default settings) is quite fond of full tablescans and parallel query. I had a lot of migrations from 8.0 to 8i were performance went bad just because of that. Now we're not interested in PQ in an OLTP, so we disabled it.
Then there's Oracle 8i favor for full tablescans. The CBO didn't have any CPU costing back then, so one of the things to do was to set optimizer_index_caching and optimizer_index_cost_adj. There have been a lot of discussing of these parameters in the past, but my experience (in 8i and on OLTP db) is that about 95% of all queries benefit and 5% of the queries will perform worse. The main problem with lowering optimizer_index_cost_adj are rounding errors in the CBO's calculation costs. This can lead to the CBO picking an index based on it's alphabetical order.
Simply explained - let's assume the cost of using index A is 9 and the cost of using index B is 5. If optimizer_index_cost_adj is set to  10, then the cost is divided by 10. So the cost of index A will be 0.9 and index B will be 0.5. Or when rounding kicks in, the cost of both A & B will be 1. And the optimizer will choose the index based on the alphabetical order of it's name. (disclaimer - simple explanation - read Jonathan Lewis' book on Oracle CBO )

<!--break-->

As expected - disabling PQ and setting the optimizer_index_x parameters - dramatically improved performance. So now I could focus on individual statements still going slow. And then I hit what I call bad database design.
Some tables were using the CHAR datatype … nothing wrong with that if you like wasting space. But the CHAR datatype was also being used for storing NUMBERs !!! The CHAR columns were indexed but that doesn't help much using where clauses like 'where <char column>=number'.

Let's setup a small testcase :

SQL> create table duh ( n number, c char(30) );
Table created.
SQL> create index i_duh_n on duh(n) ;
Index created.
SQL> create index i_duh_c on duh(c) ;
Index created.
SQL> insert into duh (n,c) select obj#,obj# from sys.obj$ ;
12201 rows created.
SQL> commit ;
Commit complete.

Querying the number column N will result in an index range scan :

SQL> select * from duh where n=2 ;

        N C
--------- ------------------------------
2 2

Execution Plan
----------------------------------------------------------
0      SELECT STATEMENT Optimizer=CHOOSE (Cost=2 Card=2 Bytes=76)
1    0   TABLE ACCESS (BY INDEX ROWID) OF 'DUH' (Cost=2 Card=2 Bytes=76)
2    1     INDEX (RANGE SCAN) OF 'I_DUH_N' (NON-UNIQUE) (Cost=1 Car d=2)

No problem here - but when we query the CHAR column C as shown below, the optimizer will perform an implicit cast, and thus no index can be used.

SQL> select * from duh where c=2 ;

        N C
--------- ------------------------------
2 2

Execution Plan
----------------------------------------------------------
0      SELECT STATEMENT Optimizer=CHOOSE (Cost=44 Card=123 Bytes=4674)
1    0   TABLE ACCESS (FULL) OF 'DUH' (Cost=44 Card=123 Bytes=4674)

All problematic queries were like the ones above. Sometimes you saw a little variation like this :

SQL> select * from duh where trim(c)='2' ;
N C
--------- ------------------------------
2 2

Execution Plan
----------------------------------------------------------
0      SELECT STATEMENT Optimizer=CHOOSE (Cost=44 Card=123 Bytes=46 74)
1    0   TABLE ACCESS (FULL) OF 'DUH' (Cost=44 Card=123 Bytes=4674)

This means the developers knew something was fishy about using CHAR as a NUMBER ;-)
Anyway - there's a quick fix for this - and it's called function based indexes (FBI). They were introduced in Oracle 8i, so, two thumbs up, I could use them. Well, I could use them after setting QUERY_REWRITE_ENABLED=TRUE and QUERY_REWRITE_INTEGRITY=TRUSTED
But I had already done this as I had seen a lot of 'WHERE UPPER(x)=UPPER(y)' type of where clauses - a developer's favorite performance killer !
So once I identified all the queries that were full table scanning these 'CHAR as NUMBER' tables, I could do the following with it :

SQL> create index i_duh_c_fbi on duh(to_number(c)) ;
Index created.
SQL> select * from duh where c=2 ;

        N C
--------- ------------------------------
2 2


Execution Plan
----------------------------------------------------------
0      SELECT STATEMENT Optimizer=CHOOSE (Cost=2 Card=2 Bytes=76)
1    0   TABLE ACCESS (BY INDEX ROWID) OF 'DUH' (Cost=2 Card=2 Bytes=76)
2    1     INDEX (RANGE SCAN) OF 'I_DUH_C_FBI' (NON-UNIQUE) (Cost=1      Card=2)

BONUS! If you know that those tables were around 60 to 80Mb each and those queries were executed quite frequently, it doesn't take a genius to see what that does for IO and performance in general. The SAN could now cope with the number of IO's thrown to it. (see the hockey stick model)

There was, however, one other thing. These tables were also queried from remote databases, mainly one XE database running the web part of the application (APEX).
After the FBI's were implemented, some parts of the web app broke down. Looking at the user dumps, we found out that the remote server process crashed each time a table (or synonym) with an fbi on it was queried. Basically we had an ORA-07445 on the XE database session each time a remote query was launched on the fbi tables :

ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_ddfnetCFull+1379] [PC:0xFD1F13] [ADDR:0x8A10068] [UNABLE_TO_READ] []

A brief search on Metalink clarified that this has something todo with 'hidden columns'.
Now ... I could only reproduce the ORA-07445 when the execution plan on the remote DB used the function based index.

So what happens if you create a function based index ? IS this really a kind of index or did the oracle kernel developers added some special trickery to the table ? Or does a function based index add a column to the table behind your back, materializing the result of the function, and indexes that ?

Let's look at the columns of table DUH without function based indexes :
SQL> select name,col#,intcol#,segcol#,property from sys.col$ where obj# = ( select obj# from sys.obj$ where name = upper('DUH') and type# = 2);

NAME                                COL#   INTCOL#   SEGCOL#  PROPERTY
------------------------------ --------- --------- --------- ---------
N                                      1         1         1         0
C                                      2         2         2         0

And let's dump a datablock of DUH (without fbi) to trace :

buffer tsn: 1 rdba: 0x00800821 (2/2081)
scn: 0x0000.00030bad seq: 0x01 flg: 0x02 tail: 0x0bad0601
frmt: 0x02 chkval: 0x0000 type: 0x06=trans data

Block header dump:  0x00800821
Object id on Block? Y
seg/obj: 0x2fce  csc: 0x00.30b73  itc: 1  flg: -  typ: 1 - DATA
fsl: 0  fnx: 0x0 ver: 0x01

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   xid:  0x0009.004.0000001f    uba: 0x00c025af.0010.0a  --U-   45  fsc 0x0000.00030bad

data_block_dump
===============
tsiz: 0x7b8
hsiz: 0x6c
pbl: 0x0372a044
bdba: 0x00800821
flag=-----------
ntab=1
nrow=45
frre=-1
fsbo=0x6c
fseo=0x137
avsp=0xcb
tosp=0xcb
0xe:pti[0] nrow=45 offs=0
0x12:pri[0] offs=0x137
0x14:pri[1] offs=0x15c
0x16:pri[2] offs=0x181
...
0x6a:pri[44] offs=0x793
block_row_dump:
tab 0, row 0, @0x137
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 02
col  1: [30]
31 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
tab 0, row 1, @0x15c
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 30
col  1: [30]
34 37 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
tab 0, row 2, @0x181
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 14
col  1: [30]
31 39 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
...
tab 0, row 44, @0x793
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 35
col  1: [30]
35 32 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
end_of_block_dump

So - nothing exciting here !

Now lets do the same after we've created a function based index on 'to_number(c)' :
SQL> select name,col#,intcol#,segcol#,property from sys.col$ where obj# = ( select obj# from sys.obj$ where name = upper('DUH') and type# = 2);

NAME                                COL#   INTCOL#   SEGCOL#  PROPERTY
------------------------------ --------- --------- --------- ---------
N                                      1         1         1         0
C                                      2         2         2         0
SYS_NC00003$                           0         3         0    327976

Hmm - a system generated column !
Property 327976 or 0x50128=0x40000+0x10000+0x00100+0x00020+0x00008
means
0x40000 = virtual column is NLS dependent
0x10000 = virtual column
0x00100 = system-generated column
0x00020 = hidden column
0x00008 = virtual column

So we have a hidden, system generated, virtual column.
COL# is zero - this is consistent with the column being hidden.
And SEGCOL# is also zero - which tells us that the column is not materialized within the table's blocks and that's consistent with the column being virtual.

You can the property values from sql.bsq in $OH/rdbms/admin :
property      number not null,           /* column properties (bit flags): */
/* 0x0001 =       1 = ADT attribute column                   */
/* 0x0002 =       2 = OID column                             */
/* 0x0004 =       4 = nested table column                    */
/* 0x0008 =       8 = virtual column                         */
/* 0x0010 =      16 = nested table's SETID$ column           */
/* 0x0020 =      32 = hidden column                          */
/* 0x0040 =      64 = primary-key based OID column           */
/* 0x0080 =     128 = column is stored in a lob              */
/* 0x0100 =     256 = system-generated column                */
/* 0x0200 =     512 = rowinfo column of typed table/view     */
/* 0x0400 =    1024 = nested table columns setid             */
/* 0x0800 =    2048 = column not insertable                  */
/* 0x1000 =    4096 = column not updatable                   */
/* 0x2000 =    8192 = column not deletable                   */
/* 0x4000 =   16384 = dropped column                         */
/* 0x8000 =   32768 = unused column - data still in row      */
/* 0x00010000 =   65536 = virtual column                         */
/* 0x00020000 =  131072 = place DESCEND operator on top          */
/* 0x00040000 =  262144 = virtual column is NLS dependent        */
/* 0x00080000 =  524288 = ref column (present as oid col)        */
/* 0x00100000 = 1048576 = hidden snapshot base table column      */
/* 0x00200000 = 2097152 = attribute column of a user-defined ref */
/* 0x00400000 = 4194304 = export hidden column,RLS on hidden col */
/* 0x00800000 = 8388608 = string column measured in characters   */
/* 0x01000000 = 16777216 = virtual column expression specified    */
/* 0x02000000 = 33554432 = typeid column                          */
/* 0x04000000 = 67108864 = Column is encrypted                    */
/* 0x20000000 = 536870912 = Column is encrypted without salt       */
         
Let's double check the column hasn't materialized by dumping a block.

buffer tsn: 1 rdba: 0x00800821 (2/2081)
scn: 0x0000.00030bad seq: 0x01 flg: 0x02 tail: 0x0bad0601
frmt: 0x02 chkval: 0x0000 type: 0x06=trans data

Block header dump:  0x00800821
Object id on Block? Y
seg/obj: 0x2fce  csc: 0x00.30b73  itc: 1  flg: -  typ: 1 - DATA
fsl: 0  fnx: 0x0 ver: 0x01

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   xid:  0x0009.004.0000001f    uba: 0x00c025af.0010.0a  --U-   45  fsc 0x0000.00030bad

data_block_dump
===============
tsiz: 0x7b8
hsiz: 0x6c
pbl: 0x0372a044
bdba: 0x00800821
flag=-----------
ntab=1
nrow=45
frre=-1
fsbo=0x6c
fseo=0x137
avsp=0xcb
tosp=0xcb
0xe:pti[0] nrow=45 offs=0
0x12:pri[0] offs=0x137
0x14:pri[1] offs=0x15c
0x16:pri[2] offs=0x181
...
0x6a:pri[44] offs=0x793
block_row_dump:
tab 0, row 0, @0x137
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 02
col  1: [30]
31 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
tab 0, row 1, @0x15c
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 30
col  1: [30]
34 37 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
tab 0, row 2, @0x181
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 14
col  1: [30]
31 39 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
...
tab 0, row 44, @0x793
tl: 37 fb: --H-FL-- lb: 0x1 cc: 2
col  0: [ 2]  c1 35
col  1: [30]
35 32 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20
end_of_block_dump

Nope - the table still has 2 physical columns. Just what we expected.

Let's dump a leaf block from the function based index... here we see the column is materialized.

buffer tsn: 1 rdba: 0x00800944 (2/2372)
scn: 0x0000.00030bc3 seq: 0x01 flg: 0x00 tail: 0x0bc30601
frmt: 0x02 chkval: 0x0000 type: 0x06=trans data

Block header dump:  0x00800944
Object id on Block? Y
seg/obj: 0x2fcf  csc: 0x00.30bbe  itc: 2  flg: -  typ: 2 - INDEX
fsl: 0  fnx: 0x0 ver: 0x01

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   xid:  0x0000.000.00000000    uba: 0x00000000.0000.00  ----    0  fsc 0x0000.00000000
0x02   xid:  0x000b.00c.00000015    uba: 0x00000000.0000.00  ----    0  fsc 0x0000.00000000

Leaf block dump
===============
header address 57843804=0x372a05c
kdxcolev 0
kdxcolok 0
kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y
kdxconco 2
kdxcosdc 0
kdxconro 117
kdxcofbo 270=0x10e
kdxcofeo 472=0x1d8
kdxcoavs 202
kdxlespl 0
kdxlende 0
kdxlenxt 8390981=0x800945
kdxleprv 0=0x0
kdxledsz 0
kdxlebksz 1892
row#0[1880] flag: -----, lock: 0
col 0; len 2; (2):  c1 02
col 1; len 6; (6):  00 80 08 21 00 00
row#1[1868] flag: -----, lock: 0
col 0; len 2; (2):  c1 03
col 1; len 6; (6):  00 80 08 21 00 19
...
row#116[472] flag: -----, lock: 0
col 0; len 3; (3):  c2 02 11
col 1; len 6; (6):  00 80 08 23 00 1a
----- end of leaf block dump -----

So a function based index is just a normal index on a virtual hidden column Wink

Now all this sounds very familiar with ... Oracle 11g virtual columns !

Let's create a table in 11g with a virtual column ... just for fun.
create table duh11g ( n number, c char(30), v as (to_number(c)) ) ;

select name,col#,intcol#,segcol#,property from sys.col$ where obj# = ( select obj# from sys.obj$ where name = upper('DUH11G') and type# = 2)
SQL> /

NAME                                 COL#    INTCOL#    SEGCOL#   PROPERTY
------------------------------ ---------- ---------- ---------- ----------
N                                       1          1          1          0
C                                       2          2          2          0
V                                       3          3          0     327688

Property 327688 = 0x50008=0x40000+0x10000+0x00008
0x40000 = virtual column is NLS dependent
0x10000 = virtual column
0x00008 = virtual column

COL# is non-zero - so it is visable. Clearly it's not system generated, but it isn't materialised because SEGCOL# is zero.
Now for fun - let's create an index on the virtual column V.

SQL> create index i_duh11g_v on duh11g( v ) ;

Index created.

And now dump a leafblock of the index :

Block header dump:  0x004159db
Object id on Block? Y
seg/obj: 0x11587  csc: 0x00.ba00b  itc: 2  flg: -  typ: 2 - INDEX
fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0006.001.000002ed  0x00c0028d.01f3.01  -B--    1  fsc 0x0000.00000000
0x02   0x0009.002.000002b7  0x00c0168b.0216.04  ----  540  fsc 0x0000.00000000
Leaf block dump
===============
header address 682344540=0x28abc05c
kdxcolev 0
KDXCOLEV Flags = - - -
kdxcolok 1
kdxcoopc 0x87: opcode=7: iot flags=--- is converted=Y
kdxconco 2
kdxcosdc 1
kdxconro 540
kdxcofbo 1116=0x45c
kdxcofeo 1116=0x45c
kdxcoavs 0
kdxlespl 0
kdxlende 0
kdxlenxt 4282844=0x4159dc
kdxleprv 0=0x0
kdxledsz 0
kdxlebksz 8032
row#0[1116] flag: ----S-, lock: 2, len=12
col 0; len 2; (2):  c1 02
col 1; len 6; (6):  00 41 59 c2 00 00
row#1[1128] flag: ----S-, lock: 2, len=12
col 0; len 2; (2):  c1 03
col 1; len 6; (6):  00 41 59 c2 00 01
...
row#539[8019] flag: ----S-, lock: 2, len=13
col 0; len 3; (3):  c2 06 29
col 1; len 6; (6):  00 41 59 c4 00 ae
----- end of leaf block dump -----


What we got is that a normal index on a virtual column in 11g is the same as a function based index in 8i.
And if you use an fbi in 8i you have actually created a hidden virtual column. It doesn't add a physical column to your table, which of course would be very io intensive and could have let to row migration (and possible chaining).
So to get back to the bug - the work around is to created a view on the table in the remote database and then query the view as opposed to querying the actual table or synonym.
This bug is also fixed in 9.2.0.6 and 10g.... but then again ... we had to work with 8.1.7.4 Wink

 

how to get viagra in adelaide

generic viagrageneric viagra [url=http://viagragche.com/]viagra coupons[/url] viagra online viagra cialis sale

viagra online order india

old get viagra cost of viagra viagra da 5 mg [url=http://viagrangk.com/#]best price for viagra[/url] wer hat viagra getestet

can you cut cialis pills in half

buy cialis brand cialis tablets cheap cialis in usa [url=http://cialiskbg.com/#]cialis tablets[/url] cialis soft tabs cheap

how to get a cash advance

need loan today loans for bad credit easy loans for bad credit [url=http://paydaymnku.com/]payday loans no credit check[/url] payday loans lakewood co

what is cheaper viagra or cialis

viagra on the cheap viagra pills viagra online aust [url=http://viagrakbg.com/#]buy viagra online[/url] viagra pill info

buy cialis viagra

viagra cialis cheap online cialis on line no pres where is the best place to buy cialis online [url=http://cialisviymw.com/#]cialis on line no pres[/url] cialis buy europe

dapoxetine and viagra together

meddirekt24 viagra generika rezeptfrei kaufen http://www.motosafi.co.ke/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=37075 cialis tablets to buy [url=http://sivivienda-ep.gob.ec/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=1294484#n]viagra pills australia[/url] cialis generic safe http://cafe-konji.de/index.php/component/k2/itemlist/user/5496 pillole per erezione cialis [url=http://wwww.bookanytime.com/index.php/component/k2/itemlist/user/122802#d]photos of viagra pills[/url] buy levitra next day delivery http://bonousa.com/?option=com_k2&view=itemlist&task=user&id=2687089 viagra 100 mg packungsbeilage [url=http://polytile.com.ua/index.php?option=com_k2&view=itemlist&task=user&id=35899#t]viagra for sale in the usa[/url] order viagra overnight shipping http://www.masters.ru/index.php?option=com_k2&view=itemlist&task=user&id=2192063 cheapest price for generic viagra [url=http://www.csw.com.my/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=1295173#b]cialis 5mg uso diario[/url] predam viagru pfizer 100mg cialis http://avord.org/index.php/component/k2/itemlist/user/274170 viagra images pill [url=http://advanced-media.co.uk/index.php/component/k2/itemlist/user/2156387#m]cialis generico su internet[/url] viagra 100mg for cheap http://restaurantcasajulian.com/?option=com_k2&view=itemlist&task=user&id=570344 cialis 5 mg nota 75 [url=http://rosttermo.ru/index.php?option=com_k2&view=itemlist&task=user&id=80126#j]wat zijn cialis pillen[/url] generic cialis pills e20 http://www.naturecare.lk/index.php?option=com_k2&view=itemlist&task=user&id=1120445 4x4 sildenafil 50 mg [url=http://nhadattrongtin.com/index.php?option=com_k2&view=itemlist&task=user&id=23516#j]buying viagra high street[/url] price tadalafil india http://udhec.com.br/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=1785577 where to buy viagra in london [url=https://gombeinvestment.com.ng/index.php/component/k2/itemlist/user/18558#r]cialis dubai price[/url] cialis 20 mg presentaciones http://microgama.com/index.php/component/k2/itemlist/user/110773 cialis on sale [url=http://www.linncountyfire.net/index.php?option=com_k2&view=itemlist&task=user&id=114683#c]buy viagra birmingham uk[/url] best viagra online pharmacy http://spyroltd.com/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=2077172 buy tadalafil uk [url=http://adiestramientocanino.com.mx/index.php/component/k2/itemlist/user/33518#h]generico de cialis en venezuela[/url]

brand viagra professional

how to prescribe viagra 100mg [url=http://viagraprofq.com/]viagra cheap[/url] generic viagra online vendo viagra sildenafil

priceline pharmacy viagra

over the counter pills that work like viagra viagra without doctor viagra 100mg preis [url=http://viagrajnmeo.com/#]viagra without a prescription[/url] viagra cuantos mg

cash advance loans

payday advance [url=https://smajloans.com/]payday loans online[/url] payday loans no credit check pay day loans [url=https://loanstrast.com/]payday loans[/url] cash advance loans payday loan online [url=https://loansfast.us.com/]online payday loans[/url] online payday loans cash advance

payday loans no credit check

cash advance [url=https://smajloans.com/]payday loans[/url] online payday loans pay day loan [url=https://loanstrast.com/]pay day loans[/url] payday advance online loans [url=https://loansfast.us.com/]online payday loans[/url] payday loan online online payday loans

soafepe

[url=http://viagrausarxonlinebuy.com/]viagra cheap[/url] buying generic viagra online http://viagrausarxonlinebuy.com viagra gelato cheap viagra online

http://paydayonlinemoneys.com

http://paydayonlinemoneys.com/ money loan with bad credit [url=http://paydayonlinemoneys.com/]payday loans online[/url] personal loans nc online payday loans

cheap cialis canadian pharmacy

cialis for sale in us cialis without a prescription buy generic viagra cialis levitra [url=http://cialisfbvne.com/#]cialis without a doctor[/url] buy cheap cialis super active

order cialis online no prescription

buy cheap cialis today cialis without doctor buy cialis tablets uk [url=http://cialisfbvne.com/#]cialis without a prescription[/url] cialis dapoxetine buy

gally

[url=http://buyviagraonlinegtrx.com/]cheap viagra[/url] viagra is used for http://buyviagraonlinegtrx.com viagra bill viagra online

le meilleur prix 20 mg cialis

buy overnight cialis [url=http://cialistas.com/]cialis[/url] buy cialis online order 100mg cialis

Aspindy

[url=http://cialisgenbuyonlinerx.com/]buy cheap cialis[/url] split cialis pills impotence http://cialisgenbuyonlinerx.com ciali cheap cialis

achat viagra en ligne france

kamagra viagra usa [url=http://viagrafzld.com/]viagra online[/url] viagra cheap viagra deutschland

Rhire

[url=http://buyviagraonlinegtrx.com/]cheap viagra online[/url] viagra nascar http://buyviagraonlinegtrx.com viagra e similares cheap viagra online

original cialis preise

we like it cialis by mail [url=http://cialismnz.com/]cheap cialis[/url] cialis farmacias cialis indias

MyronJicut

http://pharmacyorx24.com [url=http://pharmacyorx24.com]sildenafil citrate 20 mg price[/url] what is the use for sildenafil 20 mg tablets? sildenafil citrate 20 mg sildenafil 20 mg tab gree

Actiome

[url=http://viagracialisrxlevitralyrica.com/]buy generic viagra[/url] viagra 4 hours warning http://viagracialisrxlevitralyrica.com what happens if a female takes viagra buy generic viagra

viagra for sale in australia

cheap ed meds online ed medicines generic for erectile dysfunction ed medications best ed medication what are the best generic ed drugs online erectile dysfunction medications ed medications compared cheap ed medication ed medications online ed drugs online best erectile dysfunction medication in canada ed drugs meds for ed erectile dysfunction medications sold in canada erectile dysfunction drugs generic ed drugs cheap erectile dysfunction drugs-canada canadian erectile dysfunction drugs ED medication order erectile dysfunction medication generic drugs for erectile dysfunction erectile meds ed drug cost comparison buy erectile dysfunction drugs generic ed medication erectile dysfunction medication prices ed meds on line cheap ed drugs best erectile dysfunction drugs for sale online erectile dysfunction drugs online meds for erectile dysfunction generic ed meds erectile dysfunction drugs canada ed drug best ed drug best deal on ED meds cheap ed meds medication from canada ed erectile dysfunction generic name for ED drugs buy erectile dysfunction drugs online cheap erectile dysfunction medication generic erectile dysfunction medications erectile dysfunction drugs comparison impotence drugs online common drugs for ed generic impotence drugs ed meds cost effective ed drugs canadian erectile dysfunction drugs rosa impex pvt ltd viagra vs cialis vs levitra buy amoxicillin lexapro online no rx pfizer viagra 100mg price sildenafil citrate 100mg online pharmacy no prescription viagra samples 2 or 3 day shipping cialis no script

booxy

[url=http://cialisbuynorxonline.com/]generic cialis online[/url] cialis pills online generic http://cialisbuynorxonline.com buy cialis without a prescription cheap cialis

viagra more than 100 mg

cuanto sale el viagra argentina viagra prices pfizer viagra order [url=http://viagramhbfe.com/#]viagra prices[/url] viagra low price

use generic viagra

cialis cheaper canada http://agropromnika.dp.ua/index.php?option=com_k2&view=itemlist&task=user&id=1767478 where to buy viagra legally [url=http://malinavolos.ru/index.php?option=com_k2&view=itemlist&task=user&id=1318106#b]generic for sildenafil[/url] pagina segura para comprar cialis generico http://mail.cmichael.co.uk/index.php?option=com_k2&view=itemlist&task=user&id=334873 how do you get a prescription for viagra [url=http://colab.di.ionio.gr/index.php?option=com_k2&view=itemlist&task=user&id=300026#i]per comprare il viagra serve la ricetta medica[/url] can you buy viagra ebay http://advisersbitcoin.com/index.php/component/k2/itemlist/user/468291 miglior cialis generico [url=http://www.bencharlton.co.za/index.php/component/k2/itemlist/user/55474#g]taking 2 50mg viagra[/url] is the viagra online real http://www.premiumfiller.ru/index.php?option=com_k2&view=itemlist&task=user&id=263554 cheap viagra pills [url=http://www.comarsa.com.pe/index.php?option=com_k2&view=itemlist&task=user&id=777904&lang=es#j]buy cialis with money order[/url] cialis generika kaufen vorkasse http://polytile.com.ua/index.php?option=com_k2&view=itemlist&task=user&id=23205 se necesita receta medica para comprar cialis en mexico [url=http://bonousa.com/?option=com_k2&view=itemlist&task=user&id=2618512#g]can levitra viagra taken together[/url] cialis 20mg boite 4 http://www.bsonlab.com/?option=com_k2&view=itemlist&task=user&id=220973 can we buy viagra in india [url=http://environfriend.org/index.php/component/k2/itemlist/user/22975#m]viagra levitra cialis price comparison[/url] buy cialis brand http://www.labarca.pe/?option=com_k2&view=itemlist&task=user&id=23621 cialis 20mg 12 filmtabletten [url=https://www.spiruline-oi.fr/index.php/component/k2/itemlist/user/168844#w]precio cialis 5 mg[/url] safe viagra online sites http://santetoujours.info/index.php?option=com_k2&view=itemlist&task=user&id=5161224 taking 4 5mg cialis [url=http://forsageband.com/index.php/component/k2/itemlist/user/182486#y]price viagra rupees[/url] tomar cialis 10 mg http://rolamentosrs.com.br/?option=com_k2&view=itemlist&task=user&id=138508 sildenafil sandoz 100 mg hinta [url=http://ulukhar.com/index.php/component/k2/itemlist/user/821200#l]diferencia entre viagra generico y original[/url] sildenafil citrate 100mg and alcohol http://alc.develop.cinfores.com/index.php/component/k2/itemlist/user/615507 viagra buy ireland [url=http://avord.org/index.php/component/k2/itemlist/user/220817#g]can i buy viagra from overseas[/url]

is generic viagra available in united states

viagra sale china http://s454491750.onlinehome.us/?option=com_k2&view=itemlist&task=user&id=447939 cialis generico en peru [url=http://nhatmypham.com/index.php?option=com_k2&view=itemlist&task=user&id=332508#i]can you buy viagra thailand[/url] buy cialis from usa http://www.thurwach-online.de/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=493732 buy cialis online safely [url=http://www.ardawest.eu/?option=com_k2&view=itemlist&task=user&id=159389#g]compare cialis prices online[/url] cialis yearly sales http://www.multirepuestos.com.sv/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=775120 viagra 100 mg ou cialis 20mg [url=http://docesconfirma.com.br/index.php?option=com_k2&view=itemlist&task=user&id=33100&Itemid=435#z]efficacia cialis 10 mg[/url] cheap generic tadalafil http://hongduchaus.com.vn/index.php?option=com_k2&view=itemlist&task=user&id=89898 many viagra pills should take [url=http://dentaal.ru/?option=com_k2&view=itemlist&task=user&id=26459#i]who take cialis 2.5mg[/url] sildenafil citrate 100mg tab http://xn----ttbgfm0e.xn--p1ai/index.php?option=com_k2&view=itemlist&task=user&id=16114 legit cialis online [url=http://www.raceiq.us/index.php/component/k2/itemlist/user/1160805#i]viagra 50mg dose[/url] is viagra generic yet http://supplyconceptsinc.com/index.php?option=com_k2&view=itemlist&task=user&id=3693524 prices for viagra walmart [url=http://daciaenmadrid.es/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=132108#x]buy levitra in south africa[/url] cheap viagra india http://www.campinglaquercia.com/?option=com_k2&view=itemlist&task=user&id=3053 where to buy liquid viagra from [url=http://www.mohr-gartenbau.de/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=2789#q]what is cheaper cialis or viagra[/url] cialis congestione nasale http://dev2j.tjpoffice.com/?option=com_k2&view=itemlist&task=user&id=79985 costo confezione cialis 5 mg [url=http://advisersbitcoin.com/index.php/component/k2/itemlist/user/468288#h]how to get a sample pack of viagra[/url] viagra las vegas buy http://bankmitraniaga.co.id/index.php?option=com_k2&view=itemlist&task=user&id=1476189 diferencia levitra generico original [url=http://www.elektro-kamleithner.hu/ek2017/component/k2/itemlist/user/63288#m]get viagra no prescription[/url] can you buy viagra over the counter http://www.atilanews.com/index.php/component/users/?option=com_k2&view=itemlist&task=user&id=106304 where can i buy real viagra [url=http://bonousa.com/?option=com_k2&view=itemlist&task=user&id=2618540#e]shop cialis online[/url]

121 viagra cialis levitra

visit our site pfizer viagra [url=http://viagrapillskj.com/]online viagra[/url] cheap viagra venta viagra mejor generica

buy viagra sydney

viagra yellow pills viagra price online pharmacy viagra utah [url=http://viagramndet.com/#]viagra price[/url] get viagra melbourne

Areri

[url=http://viagracheapnorxonline.com/]buying viagra online[/url] which is better cialis or viagra http://viagracheapnorxonline.com discount viagra buying viagra online

urils

[url=http://cialisgenonlinerx.com/]buy cheap cialis[/url] alcohol cialis generic http://cialisgenonlinerx.com mexico city free viagra cialis pills cialis online

missouri payday loan

payday loan bad credit [url=https://loanstrast.com/]check my credit[/url] fast cash bad loans

buy cialis uk suppliers

how to take cialis pills buy cialis cialis average wholesale price [url=http://cialismbvi.com/#]cialis online[/url] discount genuine cialis

cheapest cialis from india

discount coupon for cialis cialis generic cialis pills cheap [url=http://cialisemk.com/]cialis 20mg prix en pharmacie[/url] cialis buy online uk

cialis on line itali

puedo tomar cialis con alcoho [url=http://cialisgoodss.com/]generic cialis[/url] cialis online cialis price apoteket

Veina

[url=http://buyviagrausaonlineph.com/]online viagra[/url] pfizer free viagra http://buyviagrausaonlineph.com main ingredient in viagra cheap viagra online

viagra 100mg pret

best buy on generic viagra viagra without doctor sildenafil pfizer 50 mg [url=http://viagrahukic.com/#]viagra without a doctor[/url] secure tabs online viagra 100mg

Rouse

[url=http://cialisonlinebuynorx.com/]cheap cialis online[/url] cialis 5mg online http://cialisonlinebuynorx.com buy cialis online mastercard buy cheap cialis

viagra acquisto onlin

i want to buy viagra [url=http://viagralkq.com/]buy generic viagra[/url] online viagra viagra acquisto onlin

express loans

need cash now with bad credit http://materialelectricoexpres.com/?option=com_k2&view=itemlist&task=user&id=142800 direct lenders bad credit loans [url=http://thecmccgroup.com/index.php?option=com_k2&view=itemlist&task=user&id=502693#n]houston loans[/url] emergency cash now http://www.takubundo.com/component/k2/itemlist/user/1233784 bad credit personal loans in arizona [url=http://applecrew.eu/?option=com_k2&view=itemlist&task=user&id=53757#b]3000 loan[/url] how do i get money fast http://jpnatc.org/?option=com_k2&view=itemlist&task=user&id=40475 personal loans for bad credit in michigan [url=http://tavanaei.com/?option=com_k2&view=itemlist&task=user&id=47215#f]apply for a payday loan[/url] payday loans in milwaukee http://brts03.ru/?option=com_k2&view=itemlist&task=user&id=31504 payday loan maryland [url=http://bokesol.com/?option=com_k2&view=itemlist&task=user&id=956720#b]monthly loan[/url] payday loan store milwaukee http://undercolombia.co/index.php/component/k2/itemlist/user/25007 american payday loans [url=http://www.bordigheraincomune.it/?option=com_k2&view=itemlist&task=user&id=7756#h]private lenders for personal loans[/url] quick cash loans for bad credit http://peruhipico.net/?option=com_k2&view=itemlist&task=user&id=55951 online payday loans direct lenders only [url=http://www.linkomnia.com/?option=com_k2&view=itemlist&task=user&id=801073#n]cash loan with bad credit[/url] payday loan usa http://usps.org.ua/?option=com_k2&view=itemlist&task=user&id=47488 applying for personal loans [url=http://via-augustina.org/?option=com_k2&view=itemlist&task=user&id=46485#a]apply online personal loan[/url] eagle loan company of ohio http://pittonimmobilier.fr/index.php/component/k2/itemlist/user/16676 personal loans poor credit [url=http://avtobox.com.ua/?option=com_k2&view=itemlist&task=user&id=27523#r]payday loan no credit[/url] personal loan unsecured http://www.pronostica.com.co/?option=com_k2&view=itemlist&task=user&id=12239 cash advance interest [url=http://fenyofa01.hu/?option=com_k2&view=itemlist&task=user&id=27906#k]payday loans odessa tx[/url] sameday loan http://snowboard-pfalz.de/index.php/component/k2/itemlist/user/98410 payday loans columbia mo [url=http://fmmekelle.com/?option=com_k2&view=itemlist&task=user&id=880794#w]bad credit loan companies[/url]

cialis vs viagra prices

viagra farmacia [url=http://buyviagralkq.com/]buy viagra online[/url] generic viagra online click here viagra daily



Howto's
See DUDE primer for info

Get Support

Europe

Belgium :
Kurt Van Meerbeeck
ORA600 bvba
E-mail
dude@ora600.be
Cell : +32 495 580714

Denmark :
Michael Môller
Service & Support Manager
Miracle AS
E-mail :
hra@miracleas.dk
Cell: +45 53 74 71 27


Latin America

Brazil :
HBtec
E-mail
dude@hbtec.com.br
Cell : +55 47 88497639
Contact


Africa

South Africa :
Kugendran Naidoo
NRG Consulting
E-mail
k@nrgc.co.za
Cell : +27 82 7799275